The warm-standby fast-promotion optimization (keep the in-memory cache, replay only from the standby's consistent ts) never engages, because OnLeaderStart unsubscribes all sequence groups before it reads the per-shard consistent ts that the optimization depends on — and the read filters on subscribed_, so it always returns UINT64_MAX.
Evidence
OnLeaderStart unsubscribes every sequence group early (tx_service/src/fault/cc_node.cpp:323-340):
WaitableCc sub_cc([](CcShard &ccs) {
for (auto &[grp_id, grp_info] : ccs.GetStandbysequenceGrps())
if (grp_info.subscribed_) grp_info.Unsubscribe(); // subscribed_ = false
return true;
}, core_cnt);
... sub_cc.Wait();
Later in the same function it reads the consistent ts to decide whether the cache survives (cc_node.cpp:401-425):
uint64_t last_consistent_ts = UINT64_MAX;
WaitableCc get_consistent_ts_cc([&](CcShard &ccs) {
uint64_t shard_ts = ccs.MinLastStandbyConsistentTs(); // see below
last_consistent_ts = std::min(last_consistent_ts, shard_ts);
return true;
}, ...);
...
if (last_consistent_ts != UINT64_MAX && last_consistent_ts != 0) {
cache_survivied = true;
replay_start_ts = last_consistent_ts + 1;
}
But MinLastStandbyConsistentTs only considers subscribed groups (tx_service/src/cc/cc_shard.cpp:3539-3549):
uint64_t min_ts = UINT64_MAX;
for (auto &seq_grp : standby_sequence_grps_)
if (seq_grp.second.subscribed_)
min_ts = std::min(seq_grp.second.last_standby_consistent_ts_, min_ts);
return min_ts;
Since every group was just unsubscribed, this returns UINT64_MAX on every shard, last_consistent_ts stays UINT64_MAX, cache_survivied stays false, and promotion takes ClearCcNodeGroupData(); replay_start_ts = 0 (cc_node.cpp:432-436).
Impact
Every standby promotion clears the warm cache and does a full cold recovery (KV + replay from 0), silently defeating the warm-standby fast-failover feature — slow promotions, large recovery I/O. (Whether the full-clear floor also interacts with WAL truncation to lose data on non-shared storage should be checked as a follow-on; the dead-optimization itself is confirmed.)
Fix: capture the consistent ts before unsubscribing (or have MinLastStandbyConsistentTs read a snapshot taken pre-unsubscribe).
Found during a code audit (PR #493), in the priority standby area. Verified against source at the cited lines.
🤖 Found with Claude Code
The warm-standby fast-promotion optimization (keep the in-memory cache, replay only from the standby's consistent ts) never engages, because
OnLeaderStartunsubscribes all sequence groups before it reads the per-shard consistent ts that the optimization depends on — and the read filters onsubscribed_, so it always returnsUINT64_MAX.Evidence
OnLeaderStartunsubscribes every sequence group early (tx_service/src/fault/cc_node.cpp:323-340):Later in the same function it reads the consistent ts to decide whether the cache survives (
cc_node.cpp:401-425):But
MinLastStandbyConsistentTsonly considers subscribed groups (tx_service/src/cc/cc_shard.cpp:3539-3549):Since every group was just unsubscribed, this returns
UINT64_MAXon every shard,last_consistent_tsstaysUINT64_MAX,cache_surviviedstays false, and promotion takesClearCcNodeGroupData(); replay_start_ts = 0(cc_node.cpp:432-436).Impact
Every standby promotion clears the warm cache and does a full cold recovery (KV + replay from 0), silently defeating the warm-standby fast-failover feature — slow promotions, large recovery I/O. (Whether the full-clear floor also interacts with WAL truncation to lose data on non-shared storage should be checked as a follow-on; the dead-optimization itself is confirmed.)
Fix: capture the consistent ts before unsubscribing (or have
MinLastStandbyConsistentTsread a snapshot taken pre-unsubscribe).Found during a code audit (PR #493), in the priority standby area. Verified against source at the cited lines.
🤖 Found with Claude Code