Skip to content

test(vector): the #718 backlog fixture must not race the sweep that arms the stall - #727

Merged
TinDang97 merged 1 commit into
mainfrom
fix/eh718-fixture-stall-race
Aug 25, 2026
Merged

test(vector): the #718 backlog fixture must not race the sweep that arms the stall#727
TinDang97 merged 1 commit into
mainfrom
fix/eh718-fixture-stall-race

Conversation

@TinDang97

Copy link
Copy Markdown
Collaborator

What

eh718_a_compaction_backlog_does_not_refuse_its_own_remedy (landed with #722 / moon#718) failed on Check (Windows)TRY 3 FAIL, deterministic within the run — while passing on Linux and macOS, including on its own pre-merge dispatch matrix. It is a race in the fixture, not a defect in the product, and it is currently red on main for every branch that dispatches the matrix.

Root cause

The stall arms at imm_count > --max-unflushed-immutable-segments (update_segment_stall, src/shard/segment_stall.rs:169). The test pins that threshold to 1, so the stall becomes eligible the moment round 1's FT.COMPACT produces the second immutable segment — a full round before the fixture finished loading.

The bit is set by the 1s MVCC sweep (run_mvcc_sweepupdate_segment_stall, src/shard/timers.rs:712), not by the write that created the segment. So round 2's 150 pipelined HSETs were racing that sweep:

  • fast host → round loads clean → test passes
  • Windows runner → sweep lands first → every HSET returns MOONERR busy: compaction backlog → the fixture reads it as drift and asserts

Being refused there is the fixture's goal, not its failure.

Fix (test-only)

  • Wait for the sweep after each compact; stop loading the moment the backlog exists.
  • Read a mid-round refusal as that same signal rather than as drift.
  • Round 0 stays strict — no immutable segment can exist yet, so a refusal there is a real defect (or another guard, e.g. diskfull) and still fails loudly.

Verification

step result
Reproduce on macOS (sleep 1.5s after each compact) RED — same line, same panic as Windows
Apply fix, same forced timing GREEN
Re-run without the forced sleep 5/5 green, stable
Anti-vacuity: revert moon#718 (remedies_a_segment_backlogfalse) RED, and on the right assertion: "FT.COMPACT is the command that DRAINS the backlog and it was refused BY the backlog"

Every moon#718 assertion is untouched, including the control that fails rather than skips when the stall is never reached. Side effect: the fixture stops a round earlier, 4.2s → 1.7s.

No product code changed — hence skip-changelog.

…rms the stall

`eh718_a_compaction_backlog_does_not_refuse_its_own_remedy` failed on the
Windows runner (TRY 3 FAIL, deterministic within the run) while passing on
Linux and macOS, including on its own pre-merge dispatch matrix. It is a race
in the fixture, not a defect in the product.

The stall arms at `imm_count > --max-unflushed-immutable-segments`
(`update_segment_stall`). The test pins that threshold to 1, so the stall
becomes eligible as soon as round 1's `FT.COMPACT` produces the SECOND
immutable segment — one full round before the fixture finished loading. The bit
itself is set by the 1s MVCC sweep rather than by the write that created the
segment, so round 2's 150 pipelined HSETs were racing that sweep. A fast host
won and the round loaded clean; the Windows runner lost and every HSET came
back `MOONERR busy: compaction backlog` — which the fixture read as drift and
asserted on.

Being refused there is the fixture's own goal, not its failure. The loader now
waits for the sweep after each compact and stops the moment the backlog exists,
and reads a mid-round refusal as that same signal. Round 0 stays strict: no
immutable segment can exist yet, so a refusal there is a real defect (or some
other guard firing) and still fails loudly.

Reproduced deterministically on macOS by sleeping 1.5s after each compact —
same line, same panic as Windows — then fixed and re-run 5/5 green. Verified
non-vacuous by reverting the moon#718 exemption (`remedies_a_segment_backlog`
-> false): the relaxed fixture still fails, and still on the `FT.COMPACT was
refused BY the backlog` assertion. Every moon#718 assertion is untouched,
including the control that fails rather than skips when the stall is never
reached. Side effect: the test stops loading a round earlier, 4.2s -> 1.7s.

Test-only; no product code changed.

author: Tin Dang
@qodo-code-review

Copy link
Copy Markdown

ⓘ Qodo reviews are paused because the subscription is no longer active. Ask your workspace admin to reactivate the subscription to resume reviews. Manage billing

@coderabbitai

coderabbitai Bot commented Aug 25, 2026

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 14 minutes.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 6ede50c9-f27b-4ad4-8a24-41cc344983b4

📥 Commits

Reviewing files that changed from the base of the PR and between 0d42ffa and 8ed9b0d.

📒 Files selected for processing (1)
  • tests/compaction_escape_hatch_718.rs

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@TinDang97 TinDang97 added the skip-changelog Skip the CHANGELOG.md update gate for this PR label Aug 25, 2026
@TinDang97
TinDang97 merged commit e2c66a6 into main Aug 25, 2026
32 of 33 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

skip-changelog Skip the CHANGELOG.md update gate for this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant