Skip to content

Automate the kill-9 half of #36's lease-reclaim gate - #114

Merged
stephane-segning merged 2 commits into
mainfrom
claude/start-36-17bf68
Aug 3, 2026
Merged

Automate the kill-9 half of #36's lease-reclaim gate#114
stephane-segning merged 2 commits into
mainfrom
claude/start-36-17bf68

Conversation

@stephane-segning

Copy link
Copy Markdown
Contributor

Summary

Intent

Source of truth: #36 ("Gate: real handset delivery, and kill -9 lease reclaim"), part of the M2 epic (#26). #36's own text names the risk directly: "lease reclamation is the kind of code that is easy to write and easy to never actually exercise." The runbook from #106 already covers the two genuinely physical acceptance tests (a real Orange handset, a human-timed kill -9) and says explicitly that neither can run under CI or an agent. But that runbook itself only had one undocumented manual dry run as evidence the kill -9/reclaim mechanic actually works — this PR turns that into a permanent, rerunnable automated test, closing the buildable half of #36 without claiming to close the issue itself.

Scope

  • New test spawns the actual compiled sms-worker binary as an OS subprocess (via CARGO_BIN_EXE_sms-worker, only injected for tests in the package defining the [[bin]] — why this lives in app/sms-worker, not crates/sms-worker), against a live Postgres and a wiremock-backed Orange stand-in whose submit endpoint delays its first response for 5s.
  • Sequence: seed an accepted message, spawn worker 1, poll until routed (the claim write lands before the provider HTTP call, per dispatch.rs's own tick), sleep 500ms into the delay window, send real SIGKILL (Child::kill() — no Rust destructor runs, unlike claim_live_postgres.rs's in-process leaseUntil-in-the-past simulation), assert the row is stuck routed with no providerMessageRef, force the lease into the past (same technique every other live suite here uses), spawn worker 2 (a real second process, not the same one resuming), and assert it reaches submitted with the same attempts count and a new providerMessageRef.
  • Final assertion is deliberately not "no double send": it asserts the mock's submit endpoint was hit exactly twice, matching the runbook's own documented, currently-unmitigated finding that a crash in this exact window produces a real second outbound submission (no DB-level uniqueness on providerMessageRef, no provider-side dedup key). The test is written to surface that gap permanently, not to paper over it.
  • Out of scope, same as before: the two actually-physical parts of Gate: real handset delivery, and kill -9 lease reclaim #36 (real Orange account, real handset) are untouched — this PR doesn't and can't close the issue.

Verification

  • Ran the new test against a real dockerized Postgres 16 (migrations applied via ci/apply-migrations.sh), 4 consecutive runs, all green and deterministic (~2.5s each) — not just cargo check.
  • just check and just lint (fmt + clippy -D warnings) — both green workspace-wide.
  • cargo check -p sms-worker-bin --tests and cargo build -p sms-worker-bin — both green before running the live test.
  • Cargo.lock diff is +3 lines only (wiremock already resolved elsewhere in the workspace) — no version conflicts introduced.

Screenshots/Evidence

N/A — a new backend integration test, no UI surface.

Risk Assessment

Low. Additive only: a new #[ignore]d test file (never runs in default cargo test/CI unless explicitly invoked with --ignored) and new dev-only dependencies. No production code path changes.

AI Usage Declaration

This PR was implemented by Claude Code (Claude Sonnet 5) under my direction, after researching the existing claim/dispatch/lease code and the prior #106 runbook, then writing and running the test live against a real Postgres instance repeatedly before committing.

  • I reviewed the test and understand exactly what it proves and what it deliberately does not (it is not a substitute for the real-handset and real-kill -9-against-Orange acceptance criteria in Gate: real handset delivery, and kill -9 lease reclaim #36).
  • I confirmed this was run against a live database rather than trusting a green cargo check, per this repo's own house rule that a compiling test proves nothing about runtime behavior.
  • I take responsibility for the accuracy of the AGENTS.md/runbook status updates — in particular that Gate: real handset delivery, and kill -9 lease reclaim #36 is explicitly described as still open pending the two physical tests, not closed by this PR.

Reviewer Focus

  • Whether asserting "exactly 2 submit calls" is the right shape for the final assertion, versus something more lenient — this is deliberately pinning down a known gap (no submit-side idempotency), not asserting a happy-path invariant.
  • Whether 5s is a safe-enough delay margin for CI/dev-machine variance in subprocess spawn latency, given this test isn't part of the default CI run (ignored by default) but might get picked up by a future live-suite sweep.

#36's own text is the reason: "lease reclamation is the kind of code
that is easy to write and easy to never actually exercise." The only
prior evidence the mechanic worked was one undocumented manual dry run
(referenced in docs/runbooks/36-handset-gate.md). This makes it a
permanent, rerunnable regression test instead of institutional memory.

app/sms-worker/tests/kill9_reclaim_live.rs spawns the real sms-worker
binary as a subprocess against a live Postgres and a wiremock-delayed
Orange stand-in, sends it a genuine SIGKILL mid-submit (no Rust
destructor ever runs, unlike claim_live_postgres.rs's in-process
leaseUntil-in-the-past simulation), and proves a second, independent
process reclaims and resubmits without losing the message. It also
pins down, as an assertion rather than a one-off finding, that the
resubmit is a genuinely new outbound call: providerMessageRef has no
DB-level uniqueness constraint and nothing gives Orange a dedup key
today, so a crash in this window produces two real submissions. That
gap is tracked and accepted, not hidden by the test.

Lives in app/sms-worker (the binary crate), not crates/sms-worker,
because Cargo only injects CARGO_BIN_EXE_sms-worker for integration
tests in the package that defines the [[bin]].

The two acceptance criteria in #36 itself remain genuinely manual —
real Orange handset delivery and a human-timed kill-9 against a real
account. This closes the buildable half only; the runbook is still
the actual gate for the issue.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@changeset-bot

changeset-bot Bot commented Aug 3, 2026

Copy link
Copy Markdown

⚠️ No Changeset found

Latest commit: b56dda2

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

wiremock = "0.6" was duplicated as a literal string in three crates
(sms-provider-orange-cm, sms-worker, and the new app/sms-worker dev-
dependency from the #36 kill-9 test) instead of following this
workspace's own convention of pinning once in [workspace.dependencies]
and referencing it with `wiremock.workspace = true` everywhere else.

Cargo.lock is unchanged — same version resolves either way, this is
purely about having one place to bump it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@stephane-segning
stephane-segning merged commit 66139e5 into main Aug 3, 2026
4 checks passed
@stephane-segning
stephane-segning deleted the claude/start-36-17bf68 branch August 3, 2026 12:43
stephane-segning added a commit that referenced this pull request Aug 3, 2026
#117)

Picks up cratestack/cratestack#341, which resolved the issue this repo
filed as cratestack/cratestack#334: cratestack-pg's
install_fips_crypto_provider() returned Ok(()) while installing nothing,
because crypto-aws-lc-rs was declared as an empty feature. Enabling it
is now a hard compile_error! instead of a false success. Verified the
fix is present in 0.6.7's vendored source rather than inferred from the
version, and that 0.6.0's SQLSTATE fix has not regressed alongside it
(33 cool_error_from_sqlx calls, 0 raw CoolError::Database mappings
across the twelve write paths).

We do not enable crypto-aws-lc-rs, so the compile_error! does not reach
this workspace; the bump is behaviour-neutral here.

AGENTS.md correction, which matters more than the bump. A previous
revision of this file — mine — claimed aws-lc-rs reaches this tree from
four independent paths including cratestack and sqlx, and that getting
off it was blocked upstream. That was wrong. It came from reading
`cargo tree -i aws-lc-rs`, which lists everything depending on the
shared rustls rather than everything asking for aws-lc-rs; Cargo unifies
features, so one rustls carries both providers' flags regardless of who
requested which. `cargo tree -e features -i rustls` shows the real
edges.

What the sources actually say: cratestack-sqlx selects sqlx-core's
_tls-rustls-ring-webpki, and cratestack-client-rust selects reqwest's
rustls-tls, which in reqwest 0.12 expands to __rustls-ring. Both
hard-select ring. The aws-lc-rs edge is reqwest 0.13's default features
via authkestra-engine — precisely what marcjazz/authkestra#179 made
opt-out-able in 0.3.3.

Confirmed by experiment, not argument: setting default-features = false
plus rustls-no-provider on the four authkestra pins makes
`cargo tree -i aws-lc-rs` report "did not match any packages". So a
static musl/scratch build or a cargo-deny policy banning aws-lc-rs is
reachable today with no upstream blocker. Deliberately not done in this
commit — rustls-no-provider panics at reqwest::Client::new() unless the
application installs a CryptoProvider first, and that runtime path is
unverified. It deserves a live TLS test, not a drive-by.

Also refreshed a stale line: M1 is done (#23, #24, #25 all merged,
closing #18), not still open.

Verified: cargo build --workspace, fmt --check, clippy -D warnings (0),
cargo test --workspace (198 passed), and the full live sweep against a
real Postgres 16 — 62 passed, 1 failed. That failure is #114's kill-9
gate and is not caused by this bump: it reproduces on main at 0.6.3,
passes in isolation, and its binary holds exactly one test, so it cannot
be #102's within-binary cause. Filed as #116.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant