The gap
ServiceSpec has no memory field, and the rendered systemd unit sets no MemoryMax= / MemoryHigh=. A service therefore has no memory ceiling of its own.
Consequences:
- A leaking or runaway service takes the whole host with it. There is nothing to stop one service consuming all memory on a box shared with an ingress, a mesh proxy, a database cluster and other services. The blast radius of one service's bug is every service on the host.
- Memory pressure is only ever visible as a HOST problem.
Host Memory Saturated fires and tells you the box is full; it cannot tell you which service did it, and the OOM killer's choice of victim need not be the culprit.
- There is no honest per-service memory alert to write. Without a cap, any "high memory" threshold is arbitrary — services legitimately have very different footprints, and we have no surface on which to tune one.
What to add
A memory cap on ServiceSpec, imposed on the unit:
# regional/service/<name>/manifest.yaml
memory: 512Mi # (shape TBD)
→ MemoryMax= (a hard ceiling; the cgroup OOM-kills the service rather than the host), and very likely MemoryHigh= a little below it (throttle-and-reclaim first, so a brief spike is absorbed rather than killed).
Worth deciding explicitly:
- Required or optional? A required field forces every service to think about its footprint, which is the point — but it is a breaking manifest change. An optional one leaves the hole open by default.
MemoryMax alone, or MemoryHigh + MemoryMax? The latter degrades more gracefully.- Does the same apply to the mesh proxy and the database cluster units? A cluster with a hard cap is a different (and riskier) conversation than a stateless service.
Why now
wardnet/wardnet-cloud#91 makes every service report process.memory.limit — read from its own cgroup (memory.max). Today that reads max (uncapped), so the metric observes nothing and the saturation alert stays silent.
The moment this lands, the series appears and the per-service memory-saturation alert starts working — no code change in the service, no alert rewrite. The alerting side (#228) is being written against a ratio that is deliberately dormant until this exists.
So the sequencing is: cap here → ratio becomes real → alert becomes meaningful. Until then, per-service memory alerting is limited to leak detection (unbounded growth), which is strictly weaker.
The gap
ServiceSpechas no memory field, and the rendered systemd unit sets noMemoryMax=/MemoryHigh=. A service therefore has no memory ceiling of its own.Consequences:
Host Memory Saturatedfires and tells you the box is full; it cannot tell you which service did it, and the OOM killer's choice of victim need not be the culprit.What to add
A memory cap on
ServiceSpec, imposed on the unit:→
MemoryMax=(a hard ceiling; the cgroup OOM-kills the service rather than the host), and very likelyMemoryHigh=a little below it (throttle-and-reclaim first, so a brief spike is absorbed rather than killed).Worth deciding explicitly:
MemoryMaxalone, orMemoryHigh+MemoryMax? The latter degrades more gracefully.Why now
wardnet/wardnet-cloud#91 makes every service report
process.memory.limit— read from its own cgroup (memory.max). Today that readsmax(uncapped), so the metric observes nothing and the saturation alert stays silent.The moment this lands, the series appears and the per-service memory-saturation alert starts working — no code change in the service, no alert rewrite. The alerting side (#228) is being written against a ratio that is deliberately dormant until this exists.
So the sequencing is: cap here → ratio becomes real → alert becomes meaningful. Until then, per-service memory alerting is limited to leak detection (unbounded growth), which is strictly weaker.