feat(common): opt-in order-field fold in datawarehouse combine (PMT-4302)

[jgrantr]

What

Adds an opt-in, default-off ordering mode to the datawarehouse combine.js in-batch fold, for current-state (merge-type) tables fed from sharded producers.

A field flagged "combineOrder": true in the table's dw_fields schema makes the fold keep the row with the highest value of that field per natural key (e.g. source_eid), instead of the last row by arrival. The arrival counter remains as tie-breaker. Tables without the flag keep the existing arrival-order fold byte-for-byte — locked by a regression test.

Why

item-retailer-events-to-dim runs botCount: 8 sharded by md5(eid), so two updates to one item are processed by different instances with independent checkpoints and can reach the queue out of source order. For append-only loads (f_item_change_event) that is harmless. For a current-state table (f_current_inventory_state, natural-key upsert), the fold's last-arrival-wins can resolve a key to a stale row before the SQL merge guard ever sees the fresh one — the guard protects batch-vs-table, but only the fold decides row-vs-row within a batch.

This is Change 3 (fold half) of the Zero Inventory Metric design (Sharpened Option E, PMT-4302), and one half of its reviewer Confirmation 8. The group-scoped merge guard is the other half and lands in the consumers (general#62 for Redshift; rstreams-connector-datalake needs its own — its mergeFact currently has no predicate).

Base branch

Based on feature/aws-sdk-v3-again — the AWS SDK v3 bar defines the current line (CUP Shared Services). Notes for whoever merges:

• Composes with fix(combine): preserve insert data on same-batch create+delete (RPL-6000) #249 (RPL-5795 combineRecords refactor), which targets development and will arrive here via the next dev→v3 sync (Merge development (with our current Datawarehouse code) into the AWS branch. #232-style): this change orders the rows the collapse sees; it does not touch the collapse itself. The delete/reactivate rules in combineRecords are order-relative, so they remain correct under source-EID order.
• The test file is named combine-order.test.js specifically to avoid colliding with fix(combine): preserve insert data on same-batch create+delete (RPL-6000) #249's combine.test.js.
• This branch's fold intentionally does not carry development's delete-priority fix — that belongs to the dev→v3 sync, not this PR.

Mechanics

• Schema-driven opt-in: load.js scans structure for a combineOrder field flag (same pattern as nk/sk/scd) and passes { orderFields: { [table]: field } } to combine(). Because the flag rides in the shared dw_fields entry, every loader that reads the schema folds the same way — Redshift and Databricks cannot diverge at the fold.
• Sort-line format (ordered tables only): {md5(nk)}-{48-char order key}-{9-digit arrival}{json}; default tables keep {md5(nk)}-{arrival}{json}. Sort already runs under LC_ALL=C.
• NULL sorts first: the order key pads/absents with spaces (0x20), below every eid character — so a row with no source_eid (a backfill row) always loses the fold to a live row. Asserted by test, per the design's testing plan.
• Fixes a latent bug: combine()'s opts were discarded (Object.assign called with a single argument), so dateFormat could never be overridden. No in-repo caller passed opts, so no behavior change.

Tests

common/test/datawarehouse/combine-order.test.js — 8 passing (npx mocha test/datawarehouse/combine-order.test.js):
inversion repair · backfill-loses-in-both-arrival-orders · backfill-only passthrough · explicit-null survives (null is a value, not a gap) · equal-eid tie-break · per-key independence · default mode locked · per-table scoping.

Consumption plan

1. Merge here; the fold reaches development-line consumers whenever the branches reconcile (or via cherry-pick if the 4.x line needs it before the 5.x jump — sequencing owned by CUP Shared Services alongside fix(combine): preserve insert data on same-batch create+delete (RPL-6000) #249's release tooling).
2. item-dw adds "combineOrder": true to source_eid in dw_fields/f_current_inventory_state.json (one line on item-dw#23).
3. general and data-lake-ingestion-bots bump leo-connector-common to a version containing this change.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DhcouH67CgpNGpBZ29ujbk

[ch-snyk-sa]

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues
✅	Code Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.