Bound standby replication reuse pools to prevent RSS growth

[liunyl]

Problem Statement

A TxService standby can retain many gigabytes of replication memory after a write burst exceeds the rate at which its CC shards can consume forwarded messages. The configured cache memory limit bounds resident data keys in CCMaps, but it does not bound replication messages waiting to be parsed/applied or the capacity retained by the standby request/message reuse pools. As a result, process RSS and the existing memory usage ratio can grow to orders of magnitude above the configured cache budget and remain there after traffic stops.

This was reproduced with a primary/standby/voter EloqKV topology using two CC shards on the primary and standby, a 128 MiB node memory limit, and an unlimited 64-client pure-SET workload with 4 KiB values and a one-million-key range. The client reached approximately 87k SET/s and 346 MiB/s. The standby repeatedly fell out of its replication history window and resubscribed. Its two shard memory gauges reached roughly 8.3 GiB combined, process RSS reached approximately 8.9 GiB, and the out-of-sync counters accumulated 298 events. Stopping the workload for 30 seconds did not reclaim the high-water mark. Restarting only the standby reduced RSS to approximately 238 MiB.

The observed behavior matches the current ownership model:

• standby stream batches can continue to enter the parse and shard queues without a meaningful byte limit;
• the in-flight guard protects only against integer overflow, not an operational memory budget;
• standby-specific request pools grow to the maximum observed concurrency;
• completed protobuf messages return to a shared reuse queue with their large payload capacity retained;
• reuse queues are not bounded by retained bytes or object count;
• each out-of-sync sequence group can initiate another resubscription, amplifying churn while work from prior subscription tenures is still being discarded.

The resident and dirty key gauges dropping to zero during the incident was expected: resubscription clears the obsolete standby CCMap. The inflated memory ratio was not a Grafana calculation error; it represented real anonymous resident memory outside the resident-data-key population. Dashboard clamping would hide the failure and is not an acceptable fix.

Solution

Keep standby replication memory bounded independently of the CCMap cache population. A standby must apply backpressure before admitted replication work exceeds a byte budget, promptly destroy excess recycled objects instead of retaining every high-water allocation, and discard all queued work from obsolete subscription tenures without requiring a process restart.

The bounded path must cover the complete lifetime of a forwarded message: stream admission, deferred parsing, shard routing, apply/abort, and recycling. Payload size must be part of admission accounting because a request-count limit alone cannot distinguish small commands from multi-kilobyte values.

When the standby cannot keep up, TxService may use the existing bounded primary history and out-of-sync/full-resubscription behavior. It must not trade correctness for memory: messages remain ordered per sequence group, current-tenure messages are not dropped silently, and stale-tenure messages never apply. Multiple out-of-sync notifications for the same subscription tenure should coalesce into one active resubscription rather than creating a resubscribe storm.

After traffic stops, or after an out-of-sync tenure is replaced, live backlog bytes must drain and retained reuse-pool bytes must return below their configured cap without restarting the standby. The implementation should preserve reuse for the normal steady-state working set while destroying objects above the bounded retained high-water mark.

User Stories

1. As a database operator, I want standby replication memory to remain bounded, so that a follower cannot exhaust host memory when the primary produces writes faster than it can apply them.
2. As a database operator, I want the replication budget to be independent of the resident-data-key cache budget, so that each memory population has an explicit and understandable limit.
3. As an SRE, I want a sustained overload to produce backpressure or an explicit out-of-sync transition, so that overload does not become unbounded anonymous RSS growth.
4. As an SRE, I want standby RSS to fall back below a documented retained-memory cap after a burst drains, so that recovery does not require a process restart.
5. As an SRE, I want old subscription-tenure work to be discarded promptly, so that repeated resubscriptions do not retain obsolete payloads.
6. As an SRE, I want duplicate out-of-sync signals from different sequence groups to coalesce, so that one failure does not trigger a resubscription storm.
7. As an SRE, I want observable live-backlog and retained-pool byte signals, so that I can distinguish a slow consumer from a reuse-pool high-water mark.
8. As a capacity planner, I want byte-based accounting, so that 4 KiB values and small commands do not consume the same nominal quota.
9. As a capacity planner, I want the default replication budget derived from the node memory configuration or documented explicitly, so that deployments have predictable memory overhead.
10. As a cluster administrator, I want the budget to have a safe default and an override, so that unusually high-throughput deployments can tune it deliberately.
11. As an EloqKV user, I want a synchronized standby to continue applying writes in per-sequence-group order under normal load, so that bounding memory does not compromise replica correctness.
12. As an EloqKV user, I want overload recovery to preserve the existing snapshot-plus-stream bootstrap guarantees, so that a full resubscription produces a correct standby.
13. As an EloqKV user, I want current-tenure traffic to resume automatically after backlog pressure clears, so that temporary bursts do not require operator intervention.
14. As a failover operator, I want queued messages from a former standby tenure fenced and released before promotion, so that stale replication cannot affect the promoted node.
15. As a failover operator, I want memory accounting to settle correctly after role changes, so that later replication sessions start with a clean budget.
16. As a TxService developer, I want every admitted byte to have a single ownership/accounting lifecycle, so that success, rejection, stale-term abort, parsing failure, and shutdown cannot leak or double-release budget.
17. As a TxService developer, I want bounded standby-specific pools rather than a global behavior change to all CC request pools, so that unrelated transaction paths do not regress unexpectedly.
18. As a TxService developer, I want oversized protobuf payloads destroyed instead of returned to the reusable pool, so that a rare large burst cannot permanently define steady-state RSS.
19. As a TxService developer, I want backpressure implemented without blocking a shard execution context on bthread synchronization, so that the fix cannot introduce TxProcessor deadlocks.
20. As a maintainer, I want one deterministic high-level regression test for admission, drain, resubscription, and reuse, so that future pool optimizations cannot reintroduce unbounded growth.
21. As a maintainer, I want the regression test to use a small injected byte budget rather than machine RSS timing, so that it is fast and reliable in CI.
22. As a reviewer, I want benchmark evidence for sustainable-load throughput before and after the change, so that bounded reuse does not silently remove the performance benefit of pooling.
23. As a monitoring user, I want the existing memory usage signal to continue reporting real allocation, so that severe replication memory pressure remains visible instead of being clamped at 100%.
24. As a monitoring user, I want resident/dirty key metrics to retain their CCMap semantics, so that replication-buffer memory is not misrepresented as resident data keys.

Implementation Decisions

• Treat standby replication memory as a separate bounded population from resident data keys and dirty data keys. Do not change the CCMap population definitions or hide excess allocation in the query layer.
• Introduce one standby replication memory-budget owner that covers bytes retained across stream input buffers, deferred parse work, forwarded-message objects, shard-queued requests, and recyclable protobuf messages. Accounting must span ownership transfers rather than resetting at each queue boundary.
• Base admission on bytes. A request-count ceiling may exist as a secondary safety guard, but it is insufficient as the primary bound because payload sizes vary materially.
• Apply backpressure at the earliest stream-admission boundary that can safely defer work. Do not accept an unlimited number of messages and then wait inside a shard. The mechanism must be compatible with brpc stream flow control and the TxProcessor/bthread threading model.
• Do not use a bthread mutex or condition variable across shard execution and bthread waiters. Prefer atomic budget accounting and stream-level scheduling/wakeup that never blocks CcRequest::Execute().
• Bound standby-specific request reuse separately from generic transaction request pools. Existing generic pool behavior should not be changed globally without evidence that all callers tolerate it.
• Bound the shared message recycler by retained bytes and object count. A recycled message that would exceed the pool budget, or whose retained payload capacity exceeds a documented per-object threshold, must be destroyed instead of returned to the pool. Merely clearing protobuf fields is not sufficient if capacity remains allocated.
• Make release paths exactly once. Normal apply completion, term rejection, out-of-sync control messages, parse errors, shutdown, queue abort, and failed resubscription must all release both object ownership and admitted-byte accounting.
• Fence queued work by subscription tenure. Once the active/candidate standby term changes, prior-tenure parse and apply work must be released without initializing CCMaps or buffering commands.
• Coalesce out-of-sync handling for one subscription tenure. Only one resubscription may be active for a tenure even when multiple sequence groups receive the control signal. A later valid tenure may still supersede it.
• Preserve the existing protocol outcome when the primary history buffer is exceeded: the standby may become out-of-sync and bootstrap again. This issue changes resource behavior, not the snapshot/stream correctness contract or per-sequence-group ordering.
• Add low-cardinality operational signals for current admitted bytes, retained recyclable bytes, and backpressure/rejection events. Use existing internal observability facilities; do not add node-group or subscription-term labels with unbounded lifetime.
• Define and document the default byte budget, retained-pool cap, and tuning option. Defaults should be derived from the node memory configuration or be an explicit conservative value, and must leave room for CCMaps, checkpoint work, and process overhead.
• Preserve steady-state reuse within the cap. The implementation should avoid allocating a new protobuf/request for every message when the standby is keeping up.
• No wire-format or persisted-data changes are required.

Testing Decisions

• Add one primary regression seam at the highest available level: a real primary/standby TxService integration test that exercises stream receive, deferred parsing, shard queuing, apply/abort, out-of-sync resubscription, and recycling as one lifecycle.
• Extend the existing multi-node test harness only as much as necessary to support an actual standby subscription. StandbyForward tests are prior art for forwarded-message semantics, while the existing node/cluster harness is prior art for process and role lifecycle; a shallow test of the pool container alone would not reproduce this bug.
• Make the integration test deterministic by configuring a very small byte budget and pausing or slowing standby shard consumption with a test hook. Do not assert on host-wide RSS as the primary pass/fail signal.
• Drive a mix of small messages and 4 KiB payloads until the budget is reached. Assert that admitted live bytes plus retained reusable bytes stay within the configured budget and documented bounded overhead, independent of the number of messages offered.
• Verify that the sender/receiver observes backpressure rather than silently dropping current-tenure messages. After releasing the paused consumer, assert that all admitted current-tenure messages apply in order and the live budget drains.
• Trigger out-of-sync while old-tenure messages are queued. Assert that only one resubscription starts, all obsolete work releases its accounting, and no old-tenure command applies after the new tenure begins.
• After drain/resubscription, assert that retained recyclable bytes are at or below the pool cap without restarting either node, and that a subsequent sustainable write stream reuses the pool successfully.
• Exercise every terminal ownership path visible through the integration seam: normal completion, stale-term rejection, out-of-sync control handling, and shutdown with queued work. Add a small focused accounting unit test only if concurrency arithmetic cannot be made deterministic through this one integration seam.
• Run the existing standby-forwarding, transaction, and failover tests to guard protocol ordering and role-transition behavior.
• Record an before/after benchmark under a sustainable workload. Pool bounding should not materially regress throughput or latency; investigate and document any regression greater than 5% rather than masking it with a larger default budget.
• Re-run the original overload scenario manually with 4 KiB pure writes. It is acceptable for the standby to go out-of-sync, but RSS must plateau at the documented bound and return below the retained cap after traffic stops or the tenure is replaced.

Out of Scope

• Changing resident-data-key or dirty-data-key metric semantics.
• Clamping the Grafana memory usage ratio or redefining the cache memory limit to include every process allocation.
• Increasing the node memory limit as a substitute for bounded replication memory.
• Guaranteeing that a standby can sustain every write rate accepted by a primary.
• Removing the existing bounded primary history buffer or the out-of-sync/full-bootstrap recovery protocol.
• Redesigning snapshot transfer, checkpoint scheduling, checkpoint interval metrics, or checkpoint failure classification.
• Changing the replication wire format, transaction durability, WAL ordering, or persisted storage schema.
• Globally replacing all TxService object pools.

Further Notes

• The issue was discovered while manually validating cache and checkpoint observability for tx_service #561. That PR correctly surfaced the allocation; it did not create the reuse-pool behavior.
• During the overload, resident and dirty key counts reached zero because the standby repeatedly cleared obsolete CCMaps during resubscription. After restarting the standby and using a sustainable finite write probe, resident keys and dirty/checkpoint signals recovered normally.
• A 30-second post-load observation showed RSS remaining near 8.9 GiB and shard memory allocation near 8.3 GiB. Restarting the standby immediately returned RSS to approximately 238 MiB, which distinguishes retained process allocation from storage contents.
• The existing threading invariant remains critical: request Execute() runs in shard/TxProcessor context while stream and lifecycle work may run in bthreads. The implementation must not introduce cross-context bthread blocking.