The standby's forward-sequence consistency tracking has no gap detection: a forwarded command lost mid-stream is silently treated as applied, so the standby diverges from the primary and, when promoted, recovers from a checkpoint floor that is above the missing data — the data is never replayed. This is the "command-chain gap → unrecoverable" failure mode.
The masking
CcShard::UpdateLastReceivedStandbySequenceId (tx_service/src/cc/cc_shard.cpp:3498-3499) advances the consistency marker with a plain max, with no check that seq_id == last_consistent + 1:
seq_grp_info.last_consistent_standby_sequence_id_ =
std::max(seq_id, seq_grp_info.last_consistent_standby_sequence_id_);The fields that exist to detect a gap — missing_standby_seqeunce_ids_, next_expecting_standby_sequence_id_ (standby.h:121-126) — are dead code: nothing reads them on the receive path (only declared / reset). So if forward N is lost but N+1 arrives, last_consistent jumps to N+1, marking the never-applied N as consistent.
How a forward is lost without resync
Primary side advances last_sent_seq_id on StreamWrite success (buffer-accept, not delivery):
tx_service/src/cc/cc_shard.cpp:3194-3200 (ForwardStandbyMessage):
write_succ = stream_sender_->SendStandbyMessageToNode(node_id, entry_ptr->Message());
if (write_succ) { last_sent_seq_id++; } // advanced even though delivery isn't guaranteedResend after a failure starts at last_sent_seq_id + 1 (cc_shard.cpp:3063-3071). When a stream breaks, messages already accepted into the brpc stream write buffer but not yet delivered are lost, yet last_sent has advanced past them, so they are never resent — and there is no resubscribe/resync on on_closed for this path. The standby then receives a later seq id and masks the hole via the max above.
Impact
last_consistent_standby_sequence_id_ feeds the standby consistent ts, which becomes the replay_start_ts floor at promotion. A masked gap → the missing command's effect is neither applied on the standby nor replayed from WAL after promotion → permanent silent divergence / lost committed update on a former standby that becomes leader.
Fix: make the receive path reject/track out-of-order seq ids (wire up the existing missing-sequence machinery), trigger a repair/resubscribe on a detected gap, and/or advance last_sent only on confirmed delivery.
Found during a code audit (PR #493). The masking (no gap check, dead missing-seq fields) and the send-side last_sent-on-StreamWrite-success are verified against source; the loss trigger depends on brpc stream-break dropping buffered-but-unacked writes.
🤖 Found with Claude Code
The standby's forward-sequence consistency tracking has no gap detection: a forwarded command lost mid-stream is silently treated as applied, so the standby diverges from the primary and, when promoted, recovers from a checkpoint floor that is above the missing data — the data is never replayed. This is the "command-chain gap → unrecoverable" failure mode.
The masking
CcShard::UpdateLastReceivedStandbySequenceId(tx_service/src/cc/cc_shard.cpp:3498-3499) advances the consistency marker with a plain max, with no check thatseq_id == last_consistent + 1:seq_grp_info.last_consistent_standby_sequence_id_ = std::max(seq_id, seq_grp_info.last_consistent_standby_sequence_id_);The fields that exist to detect a gap —
missing_standby_seqeunce_ids_,next_expecting_standby_sequence_id_(standby.h:121-126) — are dead code: nothing reads them on the receive path (only declared / reset). So if forwardNis lost butN+1arrives,last_consistentjumps toN+1, marking the never-appliedNas consistent.How a forward is lost without resync
Primary side advances
last_sent_seq_idonStreamWritesuccess (buffer-accept, not delivery):tx_service/src/cc/cc_shard.cpp:3194-3200(ForwardStandbyMessage):Resend after a failure starts at
last_sent_seq_id + 1(cc_shard.cpp:3063-3071). When a stream breaks, messages already accepted into the brpc stream write buffer but not yet delivered are lost, yetlast_senthas advanced past them, so they are never resent — and there is no resubscribe/resync onon_closedfor this path. The standby then receives a later seq id and masks the hole via the max above.Impact
last_consistent_standby_sequence_id_feeds the standby consistent ts, which becomes thereplay_start_tsfloor at promotion. A masked gap → the missing command's effect is neither applied on the standby nor replayed from WAL after promotion → permanent silent divergence / lost committed update on a former standby that becomes leader.Fix: make the receive path reject/track out-of-order seq ids (wire up the existing missing-sequence machinery), trigger a repair/resubscribe on a detected gap, and/or advance
last_sentonly on confirmed delivery.Found during a code audit (PR #493). The masking (no gap check, dead missing-seq fields) and the send-side
last_sent-on-StreamWrite-success are verified against source; the loss trigger depends on brpc stream-break dropping buffered-but-unacked writes.🤖 Found with Claude Code