Problem
We link jemalloc as the global allocator (crates/aisix-server/src/main.rs) but never tune it — no
tikv_jemalloc_ctl, no background_thread, no dirty_decay_ms / muzzy_decay_ms, no MALLOC_CONF.
A grep across the tree returns nothing for any of those.
With the defaults, freed pages stay in jemalloc's dirty/muzzy state and are only purged lazily on
subsequent allocation activity in the same arena. For a gateway this produces a memory ratchet:
RSS tracks (peak concurrency x payload size), and after a burst of large requests the process stays
pinned near that peak even though the live set has collapsed. Under an idle-after-burst pattern the
memory is never handed back to the OS.
Why this matters for an AI gateway specifically
The request hot path allocates and frees the body several times per request (raw bytes -> parsed
JSON -> re-serialized outbound), so the ratchet is driven directly by payload size. Two traffic
shapes make that severe:
- multimodal requests carrying inline base64 images (hundreds of KB to MB per request)
- clients replaying the full conversation history on every turn, which grows monotonically
A tenant sending one burst of multimodal traffic can leave the process resident at its peak
indefinitely. In a container with a memory limit this is an OOM-kill risk that our current metrics
would not explain, because the live set looks fine.
Prior art
Reviewing mainstream Rust gateway implementations, at least one enables jemalloc's background purge
thread explicitly at startup and additionally ships a fallback: when the target does not support
background threads, it spawns its own idle-purge thread that reads opt.dirty_decay_ms /
arenas.narenas and periodically forces a purge by writing arena.<i>.dirty_decay_ms = 0. The
stated rationale is exactly the ratchet described above — the goal is that RSS plateaus under
sustained load and falls back to idle when load subsides.
Upstream reference for the knobs: https://jemalloc.net/jemalloc.3.html
(background_thread, opt.dirty_decay_ms, opt.muzzy_decay_ms, arena.<i>.dirty_decay_ms).
Measurement gap (do this first)
Our bench harness only samples peak RSS under saturation. It has no notion of "RSS after the
load stops", which is precisely the axis this issue is about — so today we cannot tell whether we
have this problem or how bad it is.
Proposed order:
- Extend the harness with a post-load RSS decay leg: drive a large-payload burst, stop the load,
then sample RSS for N seconds and record the curve (does it fall, how far, how fast).
- Measure the current binary on that leg to establish whether the ratchet is real here.
- Only then decide the fix:
background_thread alone may be enough, or it may need explicit decay
tuning; both should be measured for throughput impact too, since a purge thread costs CPU.
Notes
- Throughput impact is expected to be neutral-to-slightly-negative; the win here is memory
behaviour, not rps. Any change should be gated on the throughput regression not exceeding noise.
- Per repo convention, before implementing, compare how at least three mainstream gateways handle
allocator purge policy and record that comparison in the PR description. Only one was reviewed
while filing this.
- Related surface: the skeleton/partial request parsing idea for large bodies attacks the same
traffic shape from the allocation-volume side rather than the page-return side.
Problem
We link jemalloc as the global allocator (
crates/aisix-server/src/main.rs) but never tune it — notikv_jemalloc_ctl, nobackground_thread, nodirty_decay_ms/muzzy_decay_ms, noMALLOC_CONF.A grep across the tree returns nothing for any of those.
With the defaults, freed pages stay in jemalloc's dirty/muzzy state and are only purged lazily on
subsequent allocation activity in the same arena. For a gateway this produces a memory ratchet:
RSS tracks (peak concurrency x payload size), and after a burst of large requests the process stays
pinned near that peak even though the live set has collapsed. Under an idle-after-burst pattern the
memory is never handed back to the OS.
Why this matters for an AI gateway specifically
The request hot path allocates and frees the body several times per request (raw bytes -> parsed
JSON -> re-serialized outbound), so the ratchet is driven directly by payload size. Two traffic
shapes make that severe:
A tenant sending one burst of multimodal traffic can leave the process resident at its peak
indefinitely. In a container with a memory limit this is an OOM-kill risk that our current metrics
would not explain, because the live set looks fine.
Prior art
Reviewing mainstream Rust gateway implementations, at least one enables jemalloc's background purge
thread explicitly at startup and additionally ships a fallback: when the target does not support
background threads, it spawns its own idle-purge thread that reads
opt.dirty_decay_ms/arenas.narenasand periodically forces a purge by writingarena.<i>.dirty_decay_ms = 0. Thestated rationale is exactly the ratchet described above — the goal is that RSS plateaus under
sustained load and falls back to idle when load subsides.
Upstream reference for the knobs: https://jemalloc.net/jemalloc.3.html
(
background_thread,opt.dirty_decay_ms,opt.muzzy_decay_ms,arena.<i>.dirty_decay_ms).Measurement gap (do this first)
Our bench harness only samples peak RSS under saturation. It has no notion of "RSS after the
load stops", which is precisely the axis this issue is about — so today we cannot tell whether we
have this problem or how bad it is.
Proposed order:
then sample RSS for N seconds and record the curve (does it fall, how far, how fast).
background_threadalone may be enough, or it may need explicit decaytuning; both should be measured for throughput impact too, since a purge thread costs CPU.
Notes
behaviour, not rps. Any change should be gated on the throughput regression not exceeding noise.
allocator purge policy and record that comparison in the PR description. Only one was reviewed
while filing this.
traffic shape from the allocation-volume side rather than the page-return side.