fix(combine): preserve insert data on same-batch create+delete (RPL-6000) by johnnyC9000 · Pull Request #249 · LeoPlatform/connectors

johnnyC9000 · 2026-08-12T14:13:36Z

Problem

A dimension/fact delete in this connector is a soft close, so it needs the target row to exist. When an entity is created and then deleted within one combine batch, the collapse in common/datawarehouse/combine.js discarded the insert's data (lastObj = data), leaving a bare tombstone that the loader wrote as a sparse row (data columns null, date/dimension FKs = 1, never closed). Verified in production — 596 rows in de_cup_prod_us.public.d_invoice. Tracked as RPL-6000.

Fix

Collapse same-key records with last-event-wins ordering, but never discard data. The decision is extracted into a pure module common/datawarehouse/combine-records.js:

insert/update → delete: the delete wins, its markers stamped onto the accumulated record → created-then-soft-closed, not a bare tombstone.
delete → insert/update: the later write wins and reactivates the entity (ES-2516, "clear the deleted flag when loading data").
two writes: deep-merge (unchanged). lone/leading delete: unchanged bare tombstone.

The extraction lets the collapse be unit-tested without importing combine.js (which pulls in leo-sdk/aws-sdk). combine.js now calls the helper; behavior is otherwise identical.

Behavior

Same-batch sequence (arrival order)	Before	After
insert, delete	bare sparse tombstone (data lost)	populated + soft-closed
delete, insert	reactivated (write wins)	reactivated (unchanged)
insert, update	deep-merged	deep-merged (unchanged)
lone delete	bare tombstone	bare tombstone (unchanged)

Tests

Adds common/test/datawarehouse/combine.test.js — 7 cases across every collapse path (there was no combine coverage before). npm test in common/ and eslint on the changed files are green.

Scope / rollout

Shared leo-connector-common; consumers pin an exact version, so this reaches a connector only when it bumps its pin — nothing floats in automatically.
First consumer: the datalake connector — its importDimension companion is Chub-Engineering/rstreams-connector-datalake#33. Together they fully close RPL-6000.
Redshift/postgres path unaffected until general bumps its pin; the same defect is latent there (short read window) and tracked as RPL-5796 (mirror fix: run the soft-close flush after the insert).

Refs: RPL-6000, RPL-5796, ES-2516.

🤖 Generated with Claude Code

Note

High Risk
Changes shared warehouse combine semantics (production data shape for create+delete batches) and introduces automated npm publishing with version commit-back; incorrect collapse or release planning could ship bad rows or wrong package versions.

Overview
Fixes RPL-6000 by changing how same-natural-key rows collapse in the datawarehouse combine path: insert/update then delete now keeps accumulated column data and applies soft-delete markers (populated soft-close instead of a sparse tombstone). Delete-then-write reactivation and plain deep-merge paths stay as before. Logic lives in new combine-records.js, wired from combine.js, with 7 unit tests in combine.test.js.

Also adds npm release automation: plan-release.mjs (conventional-commit bumps, npm stable baseline, version ceilings) and release.yaml (plan job + matrix publish with Trusted Publishing OIDC, RC tags off non-master). Adds .nvmrc (Node 22), bumps leo-connector-common to 4.1.0 and minor bumps across connector packages, and adds repository.directory on manifests for monorepo publishing. The release workflow temporarily listens on fix/rpl-5795-combine-soft-delete-ordering as well as master.

^{Reviewed by Cursor Bugbot for commit 35a3901. Bugbot is set up for automated code reviews on this repo. Configure here.}

…795) A dimension/fact delete in this connector is a soft close, so it needs the target row to exist. When an entity was created and then deleted within one combine batch, the collapse discarded the insert's data (lastObj = data), leaving a bare tombstone that the loader then wrote as a sparse row (data columns null, date/dimension FKs defaulted to 1, never closed). Fix the collapse to preserve data while keeping last-event-wins ordering: - insert-then-delete: the delete wins, but the insert's data is carried onto it, so the row is created-then-soft-closed instead of tombstoned. - delete-then-insert: the later write wins and reactivates the entity (the delete is dropped), matching ES-2516 ("clear the deleted flag when loading data" — a data load after a delete undeletes the row). The collapse decision is extracted into a pure, leo-sdk-free module (combine-records.js) so it can be unit-tested without the leo-sdk/aws-sdk require chain. Adds combine unit tests (there were none before). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

ch-snyk-sa · 2026-08-12T14:13:55Z

✅ Snyk checks have passed. No issues have been found so far.

Status	Scan Engine	Critical	High	Medium	Low	Total (0)
✅	Open Source Security	0	0	0	0	0 issues
✅	Licenses	0	0	0	0	0 issues
✅	Code Security	0	0	0	0	0 issues

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

Adds a conventional-commit-driven release workflow that discovers the nine publishable packages, resolves each one's next version from the highest stable version on npm (reconciled against package.json), packs and publishes only changed packages, and authenticates via npm Trusted Publishing (OIDC) instead of a static NPM_TOKEN. Master publishes clean semver to latest; other branches publish x.y.z-rc.<run_id> to the rc dist-tag. Also fixes the repository field on all nine package.json files (missing or using the deprecated git:// protocol), which npm provenance requires.

- leo-connector-common@5.1.0 - leo-connector-elasticsearch@3.1.0 - leo-connector-entity-table@4.1.0 - leo-connector-mongo@4.1.0 - leo-connector-mysql@3.1.0 - leo-connector-oracle@2.1.0 - leo-connector-postgres@5.1.0 - leo-connector-redshift@4.1.0 - leo-connector-sqlserver@4.1.0

cursor · 2026-08-17T18:39:25Z

+    const event = JSON.parse(readFileSync(EVENT_PATH, 'utf8'));
+    const before = event.before;
+    if (!before || /^0+$/.test(before)) return null;
+    if (!gitOrNull(['cat-file', '-e', before])) return null; // not fetched / unborn


Broken git object existence check

High Severity

git cat-file -e succeeds with empty stdout, so !gitOrNull(...) is always true and readEventBefore always returns null. On master pushes with no release tags yet, resolveRange falls back to HEAD~1..HEAD, which with --no-merges can miss the whole push (e.g. merge commits) and skip or under-bump the first releases.

^{Reviewed by Cursor Bugbot for commit 0d3dc73. Configure here.}

cursor · 2026-08-17T18:39:25Z

+
+on:
+  push:
+    branches: [master, fix/rpl-5795-combine-soft-delete-ordering]


Feature branch left in push trigger

High Severity

The push trigger still lists fix/rpl-5795-combine-soft-delete-ordering, so every push to that branch runs the full release pipeline and can publish rc packages to npm. The header comments say RCs are meant to come only from workflow_dispatch; this looks like temporary test scaffolding that would also remain on master after merge.

^{Reviewed by Cursor Bugbot for commit 0d3dc73. Configure here.}

npm already has leo-connector-common v5 prereleases published, so the existing highest-stable-version baseline would resolve new patch/minor releases as v5.x. v5 is being developed on a separate branch and isn't ready to publish yet, so cap the baseline lookup at v4 for this package and exclude it from a run entirely (rather than silently publishing into v5) if a breaking-change commit would otherwise cross the ceiling.

An earlier push accidentally triggered a real (non-dry-run) release.yaml run before the v4 version ceiling was in place, which committed leo-connector-common@5.1.0 to this branch. Reset it to 4.1.0, the version the ceiling-respecting resolver actually computes.

cursor

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

There are 3 total unresolved issues (including 2 from previous reviews).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit a05e0a6. Configure here.}

…eous bumps leo-connector-common's v5.1.0 was already corrected to v4.1.0 in a05e0a6, but the other 5 packages whose majors diverge from feature/aws-sdk-v3-again (elasticsearch, entity-table, mongo, postgres, redshift) still carried the same erroneous major-jump from the same release.yaml run before ceilings were in place. Add a VERSION_CEILINGS entry for every leo-connector-* package (prior major from development for the 6 that diverge; current major for mysql/oracle/sqlserver as a no-op guard against a future collision), then revert each affected package.json/package-lock.json to <dev major>.1.0 - the version the ceiling-respecting resolver actually computes. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

johnnyC9000 changed the title ~~fix(combine): preserve insert data on same-batch create+delete (RPL-5795)~~ fix(combine): preserve insert data on same-batch create+delete (RPL-6000) Aug 12, 2026

dawilk and others added 2 commits August 17, 2026 11:35

cursor Bot reviewed Aug 17, 2026

View reviewed changes

dawilk added 2 commits August 17, 2026 11:46

cursor Bot reviewed Aug 17, 2026

View reviewed changes

Comment thread .github/scripts/plan-release.mjs

johnnyC9000 mentioned this pull request Aug 26, 2026

RPL-6780: fan out non-natural-key deletes to real natural keys (leo-connector-common) #250

Open

jgrantr mentioned this pull request Aug 27, 2026

feat(common): opt-in order-field fold in datawarehouse combine (PMT-4302) #251

Draft

Navigation Menu

Uh oh!

fix(combine): preserve insert data on same-batch create+delete (RPL-6000) - #249

fix(combine): preserve insert data on same-batch create+delete (RPL-6000)#249
johnnyC9000 wants to merge 6 commits into
developmentfrom
fix/rpl-5795-combine-soft-delete-ordering

johnnyC9000 commented Aug 12, 2026 •

edited by cursor Bot

Loading

Uh oh!

ch-snyk-sa commented Aug 12, 2026 •

edited

Loading

Uh oh!

cursor Bot Aug 17, 2026

Uh oh!

cursor Bot Aug 17, 2026

Uh oh!

cursor Bot left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

johnnyC9000 commented Aug 12, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fix

Behavior

Tests

Scope / rollout

Uh oh!

ch-snyk-sa commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Snyk checks have passed. No issues have been found so far.

Uh oh!

cursor Bot Aug 17, 2026

Choose a reason for hiding this comment

Broken git object existence check

Uh oh!

cursor Bot Aug 17, 2026

Choose a reason for hiding this comment

Feature branch left in push trigger

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

johnnyC9000 commented Aug 12, 2026 •

edited by cursor Bot

Loading

ch-snyk-sa commented Aug 12, 2026 •

edited

Loading