Automate the kill-9 half of #36's lease-reclaim gate - #114
Merged
Conversation
#36's own text is the reason: "lease reclamation is the kind of code that is easy to write and easy to never actually exercise." The only prior evidence the mechanic worked was one undocumented manual dry run (referenced in docs/runbooks/36-handset-gate.md). This makes it a permanent, rerunnable regression test instead of institutional memory. app/sms-worker/tests/kill9_reclaim_live.rs spawns the real sms-worker binary as a subprocess against a live Postgres and a wiremock-delayed Orange stand-in, sends it a genuine SIGKILL mid-submit (no Rust destructor ever runs, unlike claim_live_postgres.rs's in-process leaseUntil-in-the-past simulation), and proves a second, independent process reclaims and resubmits without losing the message. It also pins down, as an assertion rather than a one-off finding, that the resubmit is a genuinely new outbound call: providerMessageRef has no DB-level uniqueness constraint and nothing gives Orange a dedup key today, so a crash in this window produces two real submissions. That gap is tracked and accepted, not hidden by the test. Lives in app/sms-worker (the binary crate), not crates/sms-worker, because Cargo only injects CARGO_BIN_EXE_sms-worker for integration tests in the package that defines the [[bin]]. The two acceptance criteria in #36 itself remain genuinely manual — real Orange handset delivery and a human-timed kill-9 against a real account. This closes the buildable half only; the runbook is still the actual gate for the issue. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
wiremock = "0.6" was duplicated as a literal string in three crates (sms-provider-orange-cm, sms-worker, and the new app/sms-worker dev- dependency from the #36 kill-9 test) instead of following this workspace's own convention of pinning once in [workspace.dependencies] and referencing it with `wiremock.workspace = true` everywhere else. Cargo.lock is unchanged — same version resolves either way, this is purely about having one place to bump it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This was referenced Aug 3, 2026
stephane-segning
added a commit
that referenced
this pull request
Aug 3, 2026
#117) Picks up cratestack/cratestack#341, which resolved the issue this repo filed as cratestack/cratestack#334: cratestack-pg's install_fips_crypto_provider() returned Ok(()) while installing nothing, because crypto-aws-lc-rs was declared as an empty feature. Enabling it is now a hard compile_error! instead of a false success. Verified the fix is present in 0.6.7's vendored source rather than inferred from the version, and that 0.6.0's SQLSTATE fix has not regressed alongside it (33 cool_error_from_sqlx calls, 0 raw CoolError::Database mappings across the twelve write paths). We do not enable crypto-aws-lc-rs, so the compile_error! does not reach this workspace; the bump is behaviour-neutral here. AGENTS.md correction, which matters more than the bump. A previous revision of this file — mine — claimed aws-lc-rs reaches this tree from four independent paths including cratestack and sqlx, and that getting off it was blocked upstream. That was wrong. It came from reading `cargo tree -i aws-lc-rs`, which lists everything depending on the shared rustls rather than everything asking for aws-lc-rs; Cargo unifies features, so one rustls carries both providers' flags regardless of who requested which. `cargo tree -e features -i rustls` shows the real edges. What the sources actually say: cratestack-sqlx selects sqlx-core's _tls-rustls-ring-webpki, and cratestack-client-rust selects reqwest's rustls-tls, which in reqwest 0.12 expands to __rustls-ring. Both hard-select ring. The aws-lc-rs edge is reqwest 0.13's default features via authkestra-engine — precisely what marcjazz/authkestra#179 made opt-out-able in 0.3.3. Confirmed by experiment, not argument: setting default-features = false plus rustls-no-provider on the four authkestra pins makes `cargo tree -i aws-lc-rs` report "did not match any packages". So a static musl/scratch build or a cargo-deny policy banning aws-lc-rs is reachable today with no upstream blocker. Deliberately not done in this commit — rustls-no-provider panics at reqwest::Client::new() unless the application installs a CryptoProvider first, and that runtime path is unverified. It deserves a live TLS test, not a drive-by. Also refreshed a stale line: M1 is done (#23, #24, #25 all merged, closing #18), not still open. Verified: cargo build --workspace, fmt --check, clippy -D warnings (0), cargo test --workspace (198 passed), and the full live sweep against a real Postgres 16 — 62 passed, 1 failed. That failure is #114's kill-9 gate and is not caused by this bump: it reproduces on main at 0.6.3, passes in isolation, and its binary holds exactly one test, so it cannot be #102's within-binary cause. Filed as #116. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
app/sms-worker/tests/kill9_reclaim_live.rs— a new live-Postgres regression test that spawns the realsms-workerbinary, sends it a genuineSIGKILLmid-submit, and proves reclaim without loss.app/sms-worker/Cargo.toml— new dev-dependencies (chrono,serde_json,wiremock) for that test.AGENTS.md/docs/runbooks/36-handset-gate.md— status updated to point at the new test, and a stale 'claim_live_postgres tests are flaky under the full workspace live sweep (shared-database test isolation) #102 still open' note corrected (it was fixed in ca653a1).Intent
Source of truth: #36 ("Gate: real handset delivery, and
kill -9lease reclaim"), part of the M2 epic (#26). #36's own text names the risk directly: "lease reclamation is the kind of code that is easy to write and easy to never actually exercise." The runbook from #106 already covers the two genuinely physical acceptance tests (a real Orange handset, a human-timedkill -9) and says explicitly that neither can run under CI or an agent. But that runbook itself only had one undocumented manual dry run as evidence thekill -9/reclaim mechanic actually works — this PR turns that into a permanent, rerunnable automated test, closing the buildable half of #36 without claiming to close the issue itself.Scope
sms-workerbinary as an OS subprocess (viaCARGO_BIN_EXE_sms-worker, only injected for tests in the package defining the[[bin]]— why this lives inapp/sms-worker, notcrates/sms-worker), against a live Postgres and a wiremock-backed Orange stand-in whose submit endpoint delays its first response for 5s.acceptedmessage, spawn worker 1, poll untilrouted(the claim write lands before the provider HTTP call, perdispatch.rs's owntick), sleep 500ms into the delay window, send realSIGKILL(Child::kill()— no Rust destructor runs, unlikeclaim_live_postgres.rs's in-processleaseUntil-in-the-past simulation), assert the row is stuckroutedwith noproviderMessageRef, force the lease into the past (same technique every other live suite here uses), spawn worker 2 (a real second process, not the same one resuming), and assert it reachessubmittedwith the sameattemptscount and a newproviderMessageRef.providerMessageRef, no provider-side dedup key). The test is written to surface that gap permanently, not to paper over it.kill -9lease reclaim #36 (real Orange account, real handset) are untouched — this PR doesn't and can't close the issue.Verification
ci/apply-migrations.sh), 4 consecutive runs, all green and deterministic (~2.5s each) — not justcargo check.just checkandjust lint(fmt + clippy -D warnings) — both green workspace-wide.cargo check -p sms-worker-bin --testsandcargo build -p sms-worker-bin— both green before running the live test.Screenshots/Evidence
N/A — a new backend integration test, no UI surface.
Risk Assessment
Low. Additive only: a new
#[ignore]d test file (never runs in defaultcargo test/CI unless explicitly invoked with--ignored) and new dev-only dependencies. No production code path changes.AI Usage Declaration
This PR was implemented by Claude Code (Claude Sonnet 5) under my direction, after researching the existing claim/dispatch/lease code and the prior #106 runbook, then writing and running the test live against a real Postgres instance repeatedly before committing.
kill -9-against-Orange acceptance criteria in Gate: real handset delivery, andkill -9lease reclaim #36).cargo check, per this repo's own house rule that a compiling test proves nothing about runtime behavior.kill -9lease reclaim #36 is explicitly described as still open pending the two physical tests, not closed by this PR.Reviewer Focus