Skip to content

Warm-standby cache survival on promotion is dead code: OnLeaderStart unsubscribes before reading the consistent ts → every promotion falls back to full cold recovery #504

Description

@liunyl

The warm-standby fast-promotion optimization (keep the in-memory cache, replay only from the standby's consistent ts) never engages, because OnLeaderStart unsubscribes all sequence groups before it reads the per-shard consistent ts that the optimization depends on — and the read filters on subscribed_, so it always returns UINT64_MAX.

Evidence

OnLeaderStart unsubscribes every sequence group early (tx_service/src/fault/cc_node.cpp:323-340):

WaitableCc sub_cc([](CcShard &ccs) {
    for (auto &[grp_id, grp_info] : ccs.GetStandbysequenceGrps())
        if (grp_info.subscribed_) grp_info.Unsubscribe();   // subscribed_ = false
    return true;
}, core_cnt);
... sub_cc.Wait();

Later in the same function it reads the consistent ts to decide whether the cache survives (cc_node.cpp:401-425):

uint64_t last_consistent_ts = UINT64_MAX;
WaitableCc get_consistent_ts_cc([&](CcShard &ccs) {
    uint64_t shard_ts = ccs.MinLastStandbyConsistentTs();   // see below
    last_consistent_ts = std::min(last_consistent_ts, shard_ts);
    return true;
}, ...);
...
if (last_consistent_ts != UINT64_MAX && last_consistent_ts != 0) {
    cache_survivied = true;
    replay_start_ts = last_consistent_ts + 1;
}

But MinLastStandbyConsistentTs only considers subscribed groups (tx_service/src/cc/cc_shard.cpp:3539-3549):

uint64_t min_ts = UINT64_MAX;
for (auto &seq_grp : standby_sequence_grps_)
    if (seq_grp.second.subscribed_)
        min_ts = std::min(seq_grp.second.last_standby_consistent_ts_, min_ts);
return min_ts;

Since every group was just unsubscribed, this returns UINT64_MAX on every shard, last_consistent_ts stays UINT64_MAX, cache_survivied stays false, and promotion takes ClearCcNodeGroupData(); replay_start_ts = 0 (cc_node.cpp:432-436).

Impact

Every standby promotion clears the warm cache and does a full cold recovery (KV + replay from 0), silently defeating the warm-standby fast-failover feature — slow promotions, large recovery I/O. (Whether the full-clear floor also interacts with WAL truncation to lose data on non-shared storage should be checked as a follow-on; the dead-optimization itself is confirmed.)

Fix: capture the consistent ts before unsubscribing (or have MinLastStandbyConsistentTs read a snapshot taken pre-unsubscribe).


Found during a code audit (PR #493), in the priority standby area. Verified against source at the cited lines.

🤖 Found with Claude Code

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions