[devnet-9] Gloas fork transition failed: prysm state-root divergence at slot 7203 leaves the network unable to finalize

[qu0b]

Summary

devnet-9 has not finalized since epoch 223. Participation was 95% at epoch 224 and collapsed to 56% at epoch 225 — the Gloas fork epoch — and has held at 55–63% for 180+ epochs (~24 h), with balances decaying in the leak (32.0361 → 32.0247 ETH).

This is not the devnet's planned non-finality test. Two consensus clients failed the fork transition outright; ~34% of active stake has been offline since.

Root cause is upstream in Prysm and has a fix already written — see status table below.

Full investigation, nine reports: https://panda-uploads-production.devops-539.workers.dev/panda/uploads/1dfb08/report-index.html

Why the network cannot finalize

Active set ~1,001,087 validators, 1000 per node.

cohort	validators	share
Prysm frozen at slot 7199 (307 nodes)	307,000	30.7%
Grandine (1 stuck + 9 OOM-looping)	10,000	1.0%
Prysm down / unreachable (3)	3,000	0.3%
Teku beacons unresponsive (13)	13,000	1.3%
Lighthouse (7) + Lodestar (3) down	10,000	1.0%
offline outright	343,000	34.3%
gossip/CPU degradation on live hosts	—	~2.4%

Practical ceiling ~63.3% against a 66.7% threshold. Fixing Prysm is the only lever on finality — peer scoring already isolates the stuck nodes, so stopping them would not measurably help the healthy population.

Status

Upstream — client bugs, not ours

finding	status
Prysm — upgradeToGloas writes the alpha.12 bid-field form, so the post-upgrade state root diverges. 307/310 nodes + all 4 bootnodes frozen at slot 7199.	Reported by @lodekeeper as OffchainLabs/prysm#17447; we corroborated. Fix exists: prysm#17380 — 3 files, MERGEABLE, APPROVED, idle since 2026-08-24. Blocked behind #17382 (spec bump alpha.12→alpha.14, 56 files, CHANGES_REQUESTED).
Grandine — zero-count ExecutionPayloadEnvelopesByRange { start_slot: 7200, count: 0 } livelocks sync at a fork boundary followed by empty slots. >= should be > at sync_manager.rs:755 and :523. Zero post-fork blocks imported on any host.	Newly filed by us. Present in current trunk.
Grandine — 12→30 GB allocation inside a single 5 s interval, OOM-killing all 10 hosts.	Fixed upstream today: grandine#902 (merged 10:20 UTC) fixes the regression from #900. Our build was #900's merge commit. Nothing to file.
Teku — resource exhaustion treated as consensus invalidity: an OutOfMemoryError falls through internalErrorToBeConsiderAsInvalidBlock (default true), the canonical block is added to a 500-entry invalid-root LRU, and dropDescendantsOfInvalidBlock walks the poison down the canonical chain. One failed import produced ~53,000 attestation rejections over 90 min.	Newly filed by us: Consensys-Incorporated/teku#11225. Note the repo is now Consensys-Incorporated/teku; the old path 422s on search.

Ours — config and infra

finding	fix
Watchtower replaced containers 4,694+ times across 972/1008 hosts, incl. 763 in the 75 min before the fork. Also resets RestartCount, silently truncating every crash-rate measurement.	#77
-Xmx12g on a ~4M-entry registry is the proximate trigger for the Teku canonical-block rejections above.	#77
Grandine still on the devnet-8 tag — missed by 1324aa2; the spec-sheet checklist item is still unticked.	#77
Buildoors are the only 16 GB hosts in a 1002×32 GB fleet, OOM-killed 7–27×/h, and have never produced a builder bid (builder_built=0 across 20 sampled post-fork slots). EIP-7732 is this devnet's headline feature and its external builder path has never been exercised.	#78
The documented per-node size override is implemented for hetzner but not for digitalocean — silently ignored on all 1,008 nodes.	#78
rpc-* TLS handshake failures on healthy hosts: nginx never reloads after docker-gen regenerates the vhost, so the running config predates the rpc- server block. Downstream of the container churn. Explains spamoor, erpc and powfaucet at once.	Not yet filed — belongs in ansible-collection-general (docker_nginx_proxy).
checkpointz serving 404 with all 1,008 upstreams configured and 370 healthy. Cause not established; pod requests cpu: 64m while polling 1,008 upstreams.	Needs one measurement before filing.
forky cannot parse Lighthouse fork-choice: validity: null on 3,764/4,075 nodes, first null at slot 7203 — the first post-Gloas block. Teku emits zero nulls on the same network. Genuine ePBS representation gap, not a forky bug.	Not yet filed — beacon-APIs spec question + go-eth2-client tolerance + LH/Teku consistency.

Open question

Lodestar crossed the fork cleanly on that image, so nothing is broken. But devnet-9 produced no evidence about lodestar trunk crossing Gloas, and an undocumented rollback cannot be distinguished from :unstable being pulled because it was misbehaving. #77 deliberately leaves it alone rather than reverting blind.

Clean bills of health

• Execution layer: zero invalid-block, state-root-mismatch or BAL errors in 3 h across all seven ELs on nodes that crossed the fork.
• No per-EL head lag: identical median across besu/erigon/geth/nethermind/reth on a simultaneous probe.
• Peer scoring works: stuck-peer share 0.6% (lighthouse), 3.4% (nimbus), 8.7% (lodestar), 9.2% (teku) against a 30.9% no-pruning baseline.
• Reth and nimbus-el on devnet-8 tags is documented and intentional.

Artifacts

Preserved so the Prysm repro outlives the devnet:

• slot-7199 BeaconState SSZ — 559,785,197 bytes, eth-consensus-version: fulu
• slot-7203 SignedBeaconBlock SSZ — 17,133 bytes, sha256 221520b11d7b88495ae5172e1ec4b5ab294307803ed527710f7b431ad38c37fe

Offline check: process_slots(state_7199, 7200) → compare against 0x48b23aa6481e1800658b4744299bbc845566ee1ac65161328accc9190480a88a. Prysm's /eth/v1/beacon/states/{slot}/root cannot substitute — it 404s on empty slots where Lighthouse advances through them.

https://claude.ai/code/session_01LgfRTaHjnv57ASe51yw1D6

[qu0b]

Update on the buildoor item: #78 is closed and replaced by #80.

Resizing the buildoors to 32 GB was the wrong fix — they are not validator nodes and should not need a validator-sized host. Per-container measurement showed the beacon node is 62-72% of RAM on three of the four, with every sidecar combined at ~1-2 GiB, so the fix belongs in the beacon config.

The buildoors carry no validator range (only builder_index), yet buildoor.yaml subscribed all five clients to every attestation subnet and additionally set --import-all-attestations on lighthouse — roughly 31,000 attestations per slot across 64 gossip meshes into nodes that never attest or aggregate. #80 removes those, keeps the payload-preparation flags, and overrides the teku heap for this group (-Xmx12g is sized for the fleet's 32 GB hosts and cannot fit on a 16 GB buildoor alongside the EL).

Still outstanding from #78 and not carried over: nodes.tf documents a size override that is implemented for Hetzner but has no DigitalOcean equivalent, so it is silently ignored on every devnet-9 node. Latent bug, independent of buildoor sizing, worth its own PR.

https://claude.ai/code/session_01LgfRTaHjnv57ASe51yw1D6