ops(compose): add memory limits to api and worker containers#25
Open
yuzushi-dev wants to merge 3 commits into
Open
ops(compose): add memory limits to api and worker containers#25yuzushi-dev wants to merge 3 commits into
yuzushi-dev wants to merge 3 commits into
Conversation
Prod recurring OOM was diagnosed as HOST-level (dmesg: constraint=CONSTRAINT_NONE, global_oom), because no container except minio had a memory limit. On a 15G host the idle stack already sits ~9.3G (3 worker replicas ×concurrency 2 ≈5G, neo4j ~2G, api ~1.1G) with swap chronically saturated, so any growth spike triggers a global OOM — and the kernel has repeatedly killed amber2-api-1 (its RSS grows over uptime to 3.5-5.5G). Add deploy.resources memory limits (mirroring the existing minio block) so a runaway becomes a cgroup-local kill of the offending container instead of a host-wide OOM that can, in principle, hit any container: - api: limit 4G / reservation 1.5G (normal RSS ~1.1G; catches the leak before host OOM) - worker: limit 2.5G / reservation 1.25G PER REPLICA (idle ~1.7G each) These are conservative initial ceilings — they contain the blast radius, they do NOT fix the underlying api RSS growth (still to be profiled: leak vs unbounded cache) and may need tuning against real ingestion peaks. Follow-ups: reduce worker replicas/concurrency or add host RAM; profile api-1 RSS over 24h. Not applied to prod — applying requires `docker compose up -d` (restart).
…~4.4G) 48h weekday RSS profiling: api-1 working set sits ~4.4G (peak 4423M), no fast leak within a container life. The initial 4G cap was BELOW the working set and would have crashlooped api. Raise to 6G (backstop above working set, kills a genuine runaway, no crashloop). Comment now records that the cap does NOT prevent host-OOM on a 15G host — sum of working sets exceeds RAM; real cure is +RAM or fewer worker replicas / lower CELERY_CONCURRENCY.
…ed gaps Adversarial review flagged the initial limits as containment-not-cure and, more concretely, that most services stayed uncapped and the canary overlay was unbounded. Re-sized from the 48h cgroup profiling (see ops notes): - api 6G->5.5G limit, 2G->4.5G reservation. Working set plateaus at ~4.4G; the reservation guarantees it so api (the recurring OOM victim) isn't the one squeezed under host pressure. - worker 2560M->2G limit, 1280M->1.5G reservation per replica (observed ~1.67G at CELERY_CONCURRENCY=2). 3 replicas => 6G tier ceiling. - neo4j (was uncapped): 2.5G limit / 1.5G reservation — JVM heap is 1G but the page cache is off-heap and unbounded, a real host-OOM vector. - milvus (was uncapped): 1.5G limit — segment/index cache grows with the set. - celery worker_max_memory_per_child=1.5GiB: recycle a child before it reaches the 2G container cap, so gradual growth from ingestion fan-out is reclaimed by a clean restart instead of an OOM-kill + requeue (poison-pill) loop. - canary overlay: api-canary/worker-canary were uncapped; add same ceilings with LOW reservations so a cutover window can't let the canary starve the live api. Reservations now sum to ~10.75G < 15G (no scheduler overcommit); realistic concurrent working set ~12.9G leaves ~2G headroom. postgres/redis/etcd left uncapped (each <0.5G, negligible runaway risk). Residual (NOT code, needs an ops decision): two full 5.5G stacks (live+canary) plus the worker tier still cannot all peak simultaneously on 15G — a cutover needs live workers scaled down or +host RAM. Applying these limits also requires `docker compose up -d` (recreate), not `docker restart`; confirm per the prod runbook before deploying. These changes are config-validated but NOT deploy- tested (the local mirror runs a single stack, no canary / no replicas).
Owner
Author
|
Follow-up commit
Reservations now sum ~10.75G < 15G (no overcommit); realistic concurrent working set ~12.9G leaves ~2G headroom.
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Prod (
amber-u24, 15G RAM) has recurring host-level OOM kills (dmesg: constraint=CONSTRAINT_NONE, global_oom) — no container except minio had a memory limit. The kernel repeatedly killsamber2-api-1.What 48h weekday profiling showed (updated)
A cgroup
memory.currentsampler (572 samples, 2026-07-20→22) found:restart: unless-stopped).sync_docs.py→ graph_extraction/community_summary LLM fan-out on the workers).Change
Add
deploy.resourcesmemory limits (same pattern as the existingminioblock):On a 15G host the sum of real working sets exceeds RAM (api ~4.4G + 3 workers + neo4j ~2G + milvus/pg/redis/etc). Memory limits alone cannot prevent a host-OOM when several containers peak together — they only make a single-container runaway cgroup-local and the victim predictable. The durable fix is capacity: add host RAM, or reduce worker replicas / lower
CELERY_CONCURRENCY. Pair this PR with one of those.Apply / risk
Requires
docker compose up -d api worker(container recreate → brief api downtime + in-flight Celery tasks interrupted/retried, no data loss). Not applied to prod.Validation
Valid YAML;
deploy.resourcesconfirmed (api: 6G/2G,worker: 2.5G/1.25G).