From f335f60664f0cccc949e98f729e51e6fb374ff96 Mon Sep 17 00:00:00 2001 From: Eric Moore Date: Sun, 6 Sep 2026 20:16:17 -0500 Subject: [PATCH] DEPLOY: tell Bridge what to do about memory, and what not to MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three questions the ansible role now has to answer, none of which were written down anywhere it looks. **Don't set MALLOC_ARENA_MAX in compose.** 0.3.63 caps arenas in the binary before the runtime is built, which keeps §2's zero-env contract intact — the one knob this process needs is not a CIRIS env var — and puts the setting where it cannot be lost by a stack redeploy or a new host. An explicit value still wins, and the boot log says which path ran, so it is checkable rather than assumed. **Size mem_limit from a measurement.** The 1.5GB on the US node was sized against an uncapped process holding ~1.46GB committed, of which ~1.4GB was allocator free-list and 44MB was live. Capped, the same work settles near 611MB. The runbook now gives the two commands — /api/v1/debug/memory for the allocator's own accounting, and the cgroup's memory.events for whether the limit is being enforced continuously — rather than a number that will be wrong the next time something changes. **Don't copy the cap to ciris-server.** Same pathology, different composition: its largest region is a ~706MB brk heap that an arena cap does not consolidate (CIRISServer#551). Better to say so here than to have someone try it and find a third of the win. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_012B5ebRpgmkqskVYLZ7DH67 --- DEPLOY.md | 45 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/DEPLOY.md b/DEPLOY.md index 3aef8ee..cc92a43 100644 --- a/DEPLOY.md +++ b/DEPLOY.md @@ -60,6 +60,51 @@ mode are themselves the **node's** `config:*` CEG (resolved at boot, hot-applied > with the lens node or bind-mount the lens node's `data/`. Node A's `capacity:*` > arrives **only** by the consent:replication leg below. +### Memory: what the ansible role should and should NOT set + +**Do not set `MALLOC_ARENA_MAX` in compose.** The binary caps glibc's arenas +itself, before it builds its runtime (`diag::cap_malloc_arenas`). That keeps the +zero-env contract above intact: the one knob this process needs is not a CIRIS +env var, and a deployment file is a place a fix can quietly stop being applied — +a stack redeploy, a new host, a second node stood up from the same image. + +An explicit value in the environment still wins, because glibc reads it before +we run and re-deciding underneath an operator who stated a number would make +compose and the binary disagree about the same setting. Which path ran is in the +log at boot, so it is checkable rather than assumed: + +```text +capped glibc malloc arenas (CIRISStatus#69: -847MB committed, live unchanged) +MALLOC_ARENA_MAX set in the environment — leaving the operator's value alone +``` + +**Size `mem_limit` from a measurement, not from history.** The 1.5GB the US node +carries was sized against an UNCAPPED process holding ~1.46GB committed, of +which ~1.4GB was allocator free-list and ~44MB was live. With the cap the same +work settles near ~611MB. After deploying, read the actual figure rather than +guessing: + +```sh +curl -s localhost:4253/api/v1/debug/memory | jq '{ + live: .mallinfo2.uordblks, held: .mallinfo2.fordblks, + live_fraction, rss: .proc.RssAnon, swap: .proc.VmSwap }' +``` + +Then set `mem_limit` to roughly 1.5x the settled `RssAnon`, and confirm the +cgroup is not fighting it: + +```sh +cat /sys/fs/cgroup/system.slice/docker-$(docker inspect -f '{{.Id}}' ciris-status).scope/memory.events +# `max` climbing = the limit is being enforced continuously; raise it. +``` + +Headroom returned here is not free money — it is headroom `ciris-server` needs +on the same host (CIRISServer#551), so it is worth reclaiming deliberately. + +**Do not copy the arena cap to `ciris-server` expecting this result.** Same +pathology, different composition: its largest single region is a ~706MB brk +`[heap]`, which an arena cap does not consolidate. + ### Adapter config:* (probe targets, poll cadence, CORS) — owner-authored The StatusAdapter's own config is `config:*` CEG under the `status.` namespace,