devnet-9: raise teku heap, put grandine on trunk

[qu0b]

Note

Scope change (2026-09-03): the Watchtower change below (section 1) has been dropped.
Watchtower remains enabled with --interval=900 per operator decision — client image
updates stay on for the rest of the devnet. This PR now contains only the Teku heap and
Grandine image fixes (sections 2 and 3). The investigation reference remains valid.

Summary

Three independent config fixes surfaced while investigating why devnet-9 has not finalized since epoch 223. None of these caused the fork failure (that is Prysm, OffchainLabs/prysm#17447), but two of them degraded the network and one made every measurement on it non-reproducible.

Full investigation index: https://panda-uploads-production.devops-539.workers.dev/panda/uploads/1dfb08/report-index.html

Changes

1. Watchtower — stop timer-driven container churn

ethereum_node_docker_watchtower_enabled → false, and --interval=900 → --http-api-update.

Watchtower replaced client containers 4,694+ times across 972 of 1,008 hosts during this devnet, including 763 hosts in the 75 minutes before the Gloas fork and 340 more in the six hours after, while the network was already in the leak. Because it creates a new container rather than restarting, RestartCount resets on every swap — so every crash-rate figure we collected was silently truncated, and the fleet-wide "beacon Up 2 hours" reading was just a watchtower wave.

A fork transition and a deliberate non-finality test are measurements. A fleet that swaps binaries under 300–540 nodes mid-run cannot produce a reproducible one.

Two changes rather than one, deliberately:

• enabled: false stops it now. This returns devnet-9 to the role default (ethereum_node_docker_watchtower_enabled: false, ethereum_node/defaults/main.yaml:149) — devnet-9 was explicitly opting in.
• The arg swap means that if someone re-enables it later, it is trigger-only and cannot resume polling.

This is the first half of the mechanism in #8 (feat(devnet-4): expose watchtower HTTP API for gated rollouts). The second half is deliberately omitted: #8 also adds WATCHTOWER_HTTP_API_TOKEN from a new sops secret plus a vhost and DNS entry. devnet-9's all.sops.yaml has no secret_watchtower_api_token and this PR does not create one. So the HTTP API is not reachable here — which is the intent for the rest of this devnet, but means trigger-mode is not yet usable. If you want gated rollouts on devnet-9, land #8 and port the secret/vhost half.

2. Teku — raise -Xmx from 12g to 20g (this is a P0 trigger, not perf tuning)

teku.yaml:32. This one is not a tuning nicety. Our heap setting is the proximate trigger for 12 Teku hosts rejecting canonical blocks, and for roughly 75,000 gossip attestation rejections in three hours.

The chain, established by per-host correlation:

1. -Xmx12g on a ~4M-entry validator registry causes java.lang.OutOfMemoryError: Java heap space — 5,018 occurrences on teku-besu-3 alone.
2. An OutOfMemoryError is not a RejectedExecutionException, so it falls straight through Teku's internalErrorToBeConsiderAsInvalidBlock predicate (BlockManager.java:450), which returns true by default — i.e. any internal error that is not task-queue saturation permanently marks the block invalid.
3. The block root is added to a 500-entry LRU of invalid roots, and dropDescendantsOfInvalidBlock (BlockManager.java:478) walks that poison down the canonical chain.
4. Teku then rejects every attestation voting for those canonical blocks — and keeps doing so long after the memory pressure is gone, because the LRU is evicted only by 500 newer invalid roots, never by time.

The amplification, in 30-minute buckets:

window	OOM errors	failed imports	attestation rejections
07:00	5,018	1	20,299 (onset)
07:30–08:00	0	0	32,647 more

One failed import produced ~53,000 attestation rejections over 90 minutes, with zero further memory pressure in the second half.

Raising -Xmx removes the trigger on our side. The other half — Teku treating resource exhaustion as consensus-level invalidity — is filed upstream as Consensys-Incorporated/teku#11225.

Supporting measurements: Teku sits at 14.11 GiB RSS pinned against the 12g cap on 32 GB hosts with ~14 GB free; one host logged 2,455 Java heap space errors/hour at 97% CPU; four hosts dropped every 500 ms tick with the beacon API unresponsive and the JVM at ~0.5% CPU — an event loop wedged, not busy. Lighthouse on identical hosts uses 15.96 GiB unconstrained.

⚠️ The 20g figure is an empirical inference from our own fleet, not a vendor recommendation. There is no upstream Teku guidance for a ~4M-entry registry — we looked. 20g leaves roughly 8 GB for JVM overhead plus the OS, which is inside observed headroom, but it should be validated on one host before a fleet-wide rollout.

3. Grandine — point at trunk

images.yaml:10 still carried :glamsterdam-devnet-8. The devnet-9 spec sheet mandates trunk images, and 1324aa2 ("bump images") moved nine entries to trunk but missed this one — the spec sheet's own checklist still carries - [ ] prysm + grandine trunk confirmation unticked.

There is also a functional reason to move now: grandinetech/grandine#902 (merged 2026-09-03 10:20 UTC) fixes the allocation bug that OOM-killed all ten Grandine hosts here — a flat ~12 GB plateau then 12 GB → 25–30 GB inside a single 5-second interval, caused by #900 raising PrettyBigU from 2²⁰ to 2³² and SignedExecutionPayloadEnvelope::full() filling to type-level capacity. The build we ran is #900's merge commit: it has #900 and not #902.

⚠️ Verify before rollout. ethpandaops/grandine:develop was pushed 2026-09-03 11:50 UTC, 90 minutes after #902 merged, so it very likely contains the fix — but I could not read the built commit out of the registry to confirm it. Check the version banner reports a commit at or after #902 before deploying.

Notes

Lodestar is deliberately left alone. images.yaml:5 also carries a devnet-8 tag, but unlike grandine this was not an omission — 0e07117 ("swap lodestar image", @parithosh) reverted it from :unstable at 15:56:20 UTC, 3 minutes 40 seconds before the fork, with a three-word message and no body. All 31 Lodestar hosts had already been recreated in a 10 min 21 s window beforehand, so the deploy began 34 minutes before the commit that records it.

Lodestar then crossed the fork cleanly on that image, so this is not currently hurting anything. But it means devnet-9 produced no evidence about Lodestar trunk crossing Gloas, and because the rollback is undocumented we cannot rule out that :unstable was pulled because it was misbehaving — which would be an unrecorded trunk defect. Reverting blind could re-break a cohort that is currently healthy, so I have not touched it.

@parithosh — could you note why the swap was needed? That is the one thing here only you can answer.

Testing

Config-only; no playbook was run. Applying #1 and #2 requires a restart of the affected containers, and #3 a re-pull.

https://claude.ai/code/session_01LgfRTaHjnv57ASe51yw1D6

[redpandabot]

Summary

Stops watchtower-driven container churn (verified: the role default at ethereum_node/defaults/main.yaml:149 is false, and disabling triggers cleanup that removes the watchtower container), raises Teku heap to -Xmx20g with a documented-in-PR validation caveat, and moves Grandine to the floating develop tag (valid per Docker Hub, and consistent with the other branch-tag images in the file). One issue: the comment's claimed safety property about the arg swap is factually wrong per the watchtower source actually used by the role.

Issues

• 🟡 ansible/inventories/devnet-9/group_vars/all/all.yaml:230 — --http-api-update is not trigger-only; default 24h polling resumes on re-enable — see the thread on that line

---

_{Reviewed @ c6f9da42
"Worse is better." — Richard Gabriel}

[redpandabot]

🟡 --http-api-update is not trigger-only; default 24h polling resumes on re-enable

Removing --interval does not stop polling. The role runs ghcr.io/nicholas-fedor/watchtower, and in internal/flags/flags.go (identical in containrrr/watchtower) when neither --interval nor --schedule is set the schedule falls back to @every 86400s (DefaultPollIntervalSeconds = 86400), and http-api-update does not disable the scheduler — it additionally starts the :8080 API. So the comment's promise that a future re-enable 'cannot resume polling' is false: it would silently poll once every 24h (and expose an unauthenticated update endpoint). Current state is fine since watchtower is disabled and the container gets removed, but the documented guard is weaker than claimed.