[Test] Every multi-node scenario injects one fault episode and asserts recovery once — no suite runs a second episode against the recovered cluster

[pathosDev]

Use case

Reading the fault sequence of every multi-node suite gives a consistent shape: set up, inject one fault episode, assert the cluster recovers, tear down.

ShardingFailover.test.ts
  test 1  → crash('a')                                     → assert
  test 2  → crash('c')                                     → assert
  test 3  → partition('a','c'); partition('b','c')
            heal('a','c');      heal('b','c')              → assert
  test 4  → leave('c')                                     → assert
  test 5  → crash('c')                                     → assert
DowningKeepMajority.test.ts    → partition(left, right)    → assert
LeaseMajority.test.ts          → partition(left, right)    → assert
ShardingCoordinatorRecovery    → crash('a')                → assert

Test 3 is the closest to a cycle and is still a single episode: node c is isolated from both peers and then healed, once.

No suite anywhere partitions, heals, and then partitions again. No suite crashes a node, waits for recovery, and crashes another. No suite runs a join/leave cycle repeatedly against the same cluster.

This matters because the bugs that survive a single-episode test are the ones where recovery leaves residue — a registration not cleaned up, a subscription not retired, a coordinator entry for a node that is gone, a lease not released. Recovery from the first fault looks correct because the residue is invisible until something needs the slot again. The second episode is where it shows.

The tracker already contains findings of exactly that shape: #955 (DistributedData never prunes departed replicas — onMemberRemoved is a literal no-op, so replica slots grow per address forever) and #1176 (two same-name clusters never merge after a heal — the fork is permanent until one side restarts). Both are residue-after-recovery bugs. Neither could have been caught by a single-episode test, and neither was.

Proposed shape

• A churn suite: repeated join/leave cycles against a stable core, asserting that membership returns to the same set each time and that per-node bookkeeping does not grow. The second half is the point — the assertion is on the absence of residue, not just on convergence.
• An instability suite: repeated partition/heal cycles at varying widths, asserting the cluster converges every time and reaches the same view.
• Bookkeeping assertions, not just membership assertions. After N cycles, replica slots, subscriptions, shard-home entries and watch registrations should be proportional to the current member count, not to the number of cycles. That is what would have caught [Bug] DistributedData never prunes departed replicas — onMemberRemoved is a literal no-op — so replica slots grow per address forever and a replica restarting on its old address loses increments to merge-max #955.
• Keep these off the per-commit path — they are slow by nature. The nightly slot already exists.

Acceptance

• A suite runs at least three consecutive partition/heal cycles and asserts convergence each time.
• A suite runs at least three consecutive join/leave cycles.
• Both assert that per-node bookkeeping does not grow with the cycle count.
• They run nightly, not per commit.
• The assertions are shown to catch [Bug] DistributedData never prunes departed replicas — onMemberRemoved is a literal no-op — so replica slots grow per address forever and a replica restarting on its old address loses increments to merge-max #955-shaped growth (invert the fix, watch it go red).

Verification status

Confirmed by reading. Every file in tests/multi-node/ was scanned for calls to crash, leave, partition and heal; the four suites with more than one call were opened and their fault sequences read line by line. The listing above is transcribed from those reads. No file contains a loop enclosing a fault call. Related: #955, #1176, #1023 (fault primitives this suite would use once richer faults exist).