devnet-8: move ready clients to trunk and propose Watchtower-gated delivery

[qu0b]

Summary

• move devnet-8 Grandine, Erigon, and Ethrex to the trunk tags already built by eth-client-docker-image-builder and used on devnet-9; keep Prysm pinned because it failed the NFT devnet
• keep execution, beacon, and validator outside Watchtower until progressive-delivery channels exist, including on buildoor hosts
• propose a stack-wide rollout design that preserves Watchtower as the install-once node-local updater

Runtime model

Every managed component follows a deterministic per-node OCI release-channel tag. The platform freezes a completed builder digest and advances one node's channel at a time; the existing Nicholas Fedor Watchtower pulls and replaces the container locally.

This proposal means Argo Workflows, not Argo CD. Argo CD already deploys the platform's Argo Workflows installation and would deploy the WorkflowTemplate. Argo Workflows executes durable steps as Kubernetes pods, but those pods only call the registry, Assertoor, and VictoriaMetrics APIs. The Ethereum clients remain Docker containers on external hosts. There is no SSH, remote Docker access, or per-release Ansible.

Grounded in the existing builder

The proposal now follows the actual eth-client-docker-image-builder contract:

• branches.yaml plus generate_config.py define source repository/ref and target repository/tag
• the scheduled workflow checks at minute 45 hourly and builds only when the branch-tag plus 7-character commit tag is absent
• per-architecture tags are published first, followed by the floating branch and commit-qualified multi-architecture tags
• Prysm and Nimbus are already expanded into separate beacon and validator outputs
• Docker Hub is currently authoritative; Harbor publishing is best-effort

Before automatic rollout, the existing manifest action needs a narrow hardening:

1. assemble from commit-qualified per-platform tags instead of moving per-platform tags
2. require every platform from platforms.yaml instead of succeeding with a partial set
3. annotate the OCI index with full source SHA, repository/ref, run ID, and build time
4. expose the final index digest as an action/job output
5. identify coupled beacon/validator outputs as one release set

No second builder or correctness-critical webhook is proposed. Six-hour intake reads the builder catalog and completed commit-qualified outputs, validates them, resolves them to immutable digests, and coalesces intermediate builds.

Health-gated flow

1. select and freeze the newest complete builder release set
2. advance only the canary's registry channel
3. verify the exact running digest using the Watchtower fork's read-only container-details API
4. query Assertoor for per-client online/canonical readiness and advancing finality
5. compare canary/control operational metrics in VictoriaMetrics
6. advance compatibility and fleet nodes within the availability budget, or restore touched channels to their previous digests

After the one-time normal stack deployment, releases require no playbook, host login, or inventory commit. If the workflow or health systems fail, tags remain unchanged and Watchtower finds nothing new. A 15-minute scan remains cheap; the six-hour controller policy limits digest changes and therefore restarts.

See docs/client-image-delivery.md for the exact builder gaps, bootstrap, ownership boundaries, rollout gates, security, rollback, coupled beacon/validator behavior, and failure semantics.

Current behavior

This PR is a design proposal, not the controller implementation. Until the shared builder, collection, and platform changes land, client updates remain explicit staged Ansible operations. Watchtower manages only xatu-sentry on normal hosts and xatu-sentry plus buildoor on buildoor hosts, so direct trunk tags cannot cause autonomous client churn.

Validation

• one clean commit with four changed files
• git diff --check
• Ansible syntax check
• normal/buildoor Watchtower scopes remain restricted
• verified the proposal against the current eth-client-docker-image-builder actions and platform Argo Workflows deployment
• confirmed the node updater is github.com/nicholas-fedor/watchtower

[qu0b]

Reworked and force-pushed as one clean commit. The devnet-local Python controller and timer are gone; the PR now keeps only the trunk-tag/Watchtower ownership changes and a stack-wide Argo + shared Ansible + Assertor + VictoriaMetrics delivery design. The earlier buildoor Watchtower finding remains fixed in the final diff.

[qu0b]

Revised again after the operational-simplicity concern: Watchtower remains the install-once node-local actuator. The shared control plane now advances deterministic per-node registry channel tags; it does not SSH, run per-release Ansible, or access Docker. The design also uses the Nicholas Fedor fork read-only container-details endpoint to prove the running manifest digest. I clarified that today, before those shared pieces exist, client updates remain explicit staged Ansible operations—the PR does not activate the proposed controller.

[redpandabot]

Summary

The PR trims devnet-8 Watchtower scope to non-critical tooling only, moves grandine/erigon/ethrex from the frozen glamsterdam-devnet-8 tags to the floating trunk tags devnet-9 already uses (tags verified to exist upstream), and adds a forward-looking design doc for stack-wide progressive client delivery. Config edits are internally consistent, match the stated intent, and the doc's concrete claims about the Nicholas Fedor Watchtower fork check out against the fork's source; I found no blockers.

Issues

• 🟡 ansible/inventories/devnet-8/group_vars/all/images.yaml:10 — un-pinning to trunk tags while unwatching clients means reprovisioned hosts silently pull untested trunk — see the thread on that line

---

_{Reviewed @ 97afdd66
"Programs must be written for people to read, and only incidentally for machines to execute." — Abelson & Sussman}

[redpandabot]

🟡 un-pinning to trunk tags while unwatching clients means reprovisioned hosts silently pull untested trunk

grandine/erigon/ethrex move from the immutable glamsterdam-devnet-8 tags to floating develop/main in the same PR that removes execution/beacon/validator from Watchtower. With ethpandaops.general's pull-always + image_name_mismatch:recreate semantics (ethereum_node_images_always_pull: true here), any host rebuild/reprovision or images.yaml ref change now resolves the current trunk digest with none of the staged gates this PR exists to establish, undercutting the doc's 'failure is closed / client updates stay explicit' framing. It matches devnet-9, so likely acceptable, but worth confirming the interim absence of any digest pin is intended.

[qu0b]

Clarification and builder grounding added: the design means Argo Workflows, not Argo CD. Argo CD deploys the already-present Argo Workflows installation and proposed WorkflowTemplate; workflow pods coordinate external APIs while Watchtower remains the Docker-host actuator. I also audited eth-client-docker-image-builder and revised the artifact contract around its real branch/short-SHA/per-platform tags, including the required manifest hardening before those outputs are safe to promote.

[redpandabot]

Summary

Moves devnet-8 Grandine/Erigon/Ethrex onto the same floating trunk tags as devnet-9, drops execution/beacon/validator from the Watchtower scope on normal and buildoor hosts, and adds docs/client-image-delivery.md sketching a digest-channel, Watchtower-actuated progressive rollout. The tags match devnet-9, the Watchtower scopes are restricted as described, the fork endpoint claim in the doc is accurate, and nothing here blocks merge. One cross-cutting concern about non-deterministic client state during the interim stands.

Issues

• 🟡 ansible/inventories/devnet-8/group_vars/all/images.yaml:10 — floating trunk tags with no convergence mechanism create version skew on live nodes — devnet-9 pairs these exact trunk tags with Watchtower watching execution/beacon/validator (verified in devnet-9 group_vars/all/all.yaml and buildoor.yaml), so all its nodes converge on the newest digest within ~15 min. After this PR devnet-8 runs the same floating tags with no updater: every manual Ansible run and every later host rebuild silently pulls whichever newer trunk digest the tag points at that day, so peers drift apart with nothing recording which digest each node runs — directly contradicting the doc's own core rule ('New or rebuilt host: seed its deterministic channel with the accepted digest'). The doc does state this interim is intentional, so this may be accepted risk, but pinning the three digests (or at least leaving the pre-fork pinned tags in place) until the channel mechanism lands would remove the skew at no cost.

---

_{Reviewed @ 578bb297
"Programs must be written for people to read, and only incidentally for machines to execute." — Abelson & Sussman}