docs: chapter 50, lifecycle, and push delivery via aggregators - #63
Closed
ExtraToast wants to merge 3 commits into
Closed
docs: chapter 50, lifecycle, and push delivery via aggregators#63ExtraToast wants to merge 3 commits into
ExtraToast wants to merge 3 commits into
Conversation
Replaces pull-based GitOps delivery with a push-based one gated by relationship-scoped aggregators. ADR-0019 records the decision and what it costs; chapter 50 describes the pipeline. The stages: a service repo merges and publishes its Intent Fragment, no PR. Any publish triggers composition automatically, no PR. Renovate bumps the composed lock pin in each aggregator. The aggregator's PR provisions a vcluster, applies the whole composed estate and runs that relationship's suite. Green merges. On merge the aggregator applies its own slice to production. An in-cluster CronJob re-applies that slice hourly. THE BOUNDARY WAS ALREADY MEASURED. Chapter 30 split coverage into 364 intent-derived, 41 pack-delivered and 45 authored objects for entirely different reasons. Delivery splits on the same line, and class B cannot move for a concrete reason: 18 of those 41 objects are HelmRelease, and applying a HelmRelease with kubectl accomplishes nothing without Flux's helm-controller. Those 18 charts are vault, vault-secrets-operator, metrics-stack, traefik x2, cert-manager, external-dns, metallb and all observability -- the Secret Store, the mechanism that delivers secrets, Prometheus, every route, TLS, DNS and load-balancer addresses. Rendering those charts to plain manifests was considered and rejected: it means owning 18 upstream charts' values, hooks and CRD upgrade paths, and vault plus VSO are in the set, so a bad render breaks secret delivery for everything. RESOLVES CLASS C. The 45 Grafana objects divide by nature: 14 service-specific dashboards become Assets on the owning service, 3 runtime-family dashboards ship with the Runtime Profile alongside the OTEL and Pyroscope values it already injects, 14 platform dashboards ship in the observability pack, and service-overview plus service-template become derived per service from scrape, runtime and exposure. service-template becomes the renderer's template rather than a cluster object, which is what its name always implied. Coverage can now reach 100%, which matters because the ledger's own header says every entry is a deferred fix rather than a permanent exemption. What is given up, stated plainly. Pruning: Flux prunes because a Kustomization keeps an inventory; kubectl keeps none, so every applied object carries a deployer label and the delete pass is the difference between that query and the render. The cluster is the inventory, which means nothing to lose, nothing to resync, and no first-run special case -- on adoption nothing is labelled, so the delete pass deletes nothing and adoption leaves orphans for the coverage assertion instead. Continuous reconciliation: replaced by an in-cluster CronJob, not a GitHub Actions schedule. CLAUDE.md measures why -- four to seven runs per repo per day regardless of the declared interval, crons late by hours, and never build anything needing prompt reaction on a schedule alone. Drift correction is exactly a reaction to something having gone wrong. A cluster CronJob is also the pattern vault-metrics-token-renewal already uses. A single answer to what is live: given up deliberately. Each slice sits at whatever lock its aggregator merged. Forcing convergence would let one red aggregator halt the estate, which is the coupling aggregators exist to break. Lag is measured instead, and cross-slice removals go through expand/contract -- checkable by composition because it sees both sides of every inbound derivation. E_CONTRACT_TOO_EARLY covers the auth-api CORS case where removing an origin while it is still deployed at an older lock breaks production. One property improves: server-side apply reports a field-ownership conflict when a human edited a field the aggregator owns, rather than silently reverting it as continuous reconciliation would. Four new invariants bring chapter 40 to 26: E_NO_DEPLOYER, E_MULTIPLE_DEPLOYERS, E_CLASS_BOUNDARY_CONFLICT, E_CONTRACT_TOO_EARLY. And chapter 40's open items 1 and 2 are resolved -- fragments publish on merge independently of any image release, and composition runs automatically on any publish. ADRs 0006, 0014 and 0015 are revised in place, all still status: proposed. ADR-0015's rationale shifts rather than breaking: a merge is now required to deploy, but it is per-relationship rather than estate-wide and exists to run tests rather than to record pointers. Composition still requires none. Most of the test stage already exists in deploy-harness -- provision-vcluster, apply-candidate with server-side apply per DAG layer, the SC-6 triple-digest assertion, scoped Vault AppRoles, CrashLoop and VSO verification, sharded execution with retry and transient classification, an owner-approved quarantine registry, guaranteed teardown. It is wired to one central compose gate rather than to N aggregators, so the work is rewiring rather than writing.
ExtraToast
added
type: docs
…revisions An audit found the docs had drifted behind the model. Fixed. THE GLOSSARY WAS MATERIALLY STALE. It had been appended to but never revised, so it still described superseded decisions: - Claim Mode defined four modes -- env, fetch, file, write -- which the model no longer has. Replaced by two orthogonal terms: Access Tier (read, self-renew, self-roll, custody) and Delivery (env, file, self). - Contended Value conflated two things chapter 20 separated. Split into Contended Value, drawn from a finite pool and assigned, and Identity Value, unique but not pooled, declared and checked. That distinction is what lets a Service Id and an exposure name be declared at all. - Rotation Tolerance still said reload is achievable only under fetch. It is delivery: self. - Exposure still said it carries no hostname. It carries a name, which is an Identity Value; the hostname is assembled from it. - Reconcile Unit had one consumer. It now has two: a Flux Kustomization for the foundation, and the apply order for everything an Aggregator pushes. Seven terms were missing entirely: Purity Rule, Placeholder, Composed Intent, Composition Lock, Participant, Aggregator, Deployer. 39 terms to 47. A NAMING CONFLICT I INTRODUCED. Chapter 50 declared kind: Aggregator while ADR-0014 and chapter 40 said SystemTestProject. Unified on Aggregator, because the thing deploys now and a test-only name understates it. SystemTestProject is recorded under _Avoid_. THREE ENV EXAMPLES REFERENCED A FILE THAT NO LONGER EXISTS. Their comments pointed at secrets.yml, which was folded back into service.yml two revisions ago. Now they name the level the grant sits at, which is the thing a reader actually needs. Also: the decision register described ADR-0005 as "Claims by Vault path; four Claim Modes", and chapter 10's examples table cited only one of knowledge's two env files. Audit now clean: no broken relative links, no stale vocabulary, 7 example files all parsing and all referenced, 8 mermaid blocks and 24 yaml blocks valid, every ADR present in the register.
Adds the parts the specification described but never showed: the pipeline as runnable workflows, the Aggregator as a real declaration, the two Deliverables chapter 50 introduces, a negative fixture, and the setup and adoption path. CHAPTER 60, setup and adoption. Bootstrap order, onboarding a Service, creating an Aggregator, and adopting the ~30 live Services. The adoption section is the one that matters, because getting it wrong deletes production. Flux prunes, so removing a Service's manifests from the Flux source before the Aggregator owns them deletes it. The safe order sets spec.prune: false on the covering Flux Kustomization FIRST, then applies from the Aggregator, then removes the manifests, then restores prune. Steps 2 and 4 the wrong way round is the destructive case, and it is called out as such. CLAUDE.md adds a timing constraint on top: Flux reads deploy/production which a workflow publishes from main, and reconciling before the publish lands silently applies the previous revision -- which would restore the manifests just removed. Adoption order across the estate is dependency-first: nodes, data, auth, knowledge, agents, media, remainder. auth is third rather than last on purpose, because it has by far the best test coverage and is where the pipeline gets proven. media is late for the opposite reason: zero test classes means its gate cannot tell you much, so it should run on machinery already trusted. FOUR WORKFLOWS, each ONE job with many steps. CLAUDE.md measured why: 561 minutes of real compute billed 2,845, four fifths of it rounding, and prefer one job with many steps. Every step carries if: !cancelled() so a single failure still surfaces the rest. - service-publish-fragment.yml fires on merge to platform/**, not on a version tag, because an intent-only change produces no image and ADR-0019 requires it to reach an Aggregator. It pushes, resolves the digest afterwards -- an artefact cannot contain its own digest -- then pulls the package back and checks it against its own manifest. - compose.yml is triggered by the publish rather than polled, because CLAUDE.md forbids putting anything needing prompt reaction on an Actions schedule. The nightly cron is a safety net, labelled as one. It also applies a negative fixture on every run, because an assertion that stopped running looks identical to one that passes -- which is exactly how E_ROUTE_AUTH_MODE_NOT_IN_TIER came to be implemented and vacuous. - aggregator-gate.yml asserts the triple digest BEFORE provisioning, so a tampered candidate costs nothing, and applies the whole composed estate with Flux for class B and SSA for class A, so the test target mirrors production's split rather than diverging from it. - aggregator-deploy.yml prunes by label query, applies in DAG order, reads the state back rather than trusting the exit code, and reports lag. Its break-glass dispatch input documents that the annotation makes a rollback stick rather than being undone by the CronJob. NEW EXAMPLES. The Aggregator declaration with exercises many-to-many and deploys one-to-one; a renovate.json custom manager for the single pin; the generated deployer RBAC that turns deploy authority into an API-server control rather than a CI convention; the in-cluster reapply CronJob; and a negative fixture for E_DUPLICATE_SERVICE_ID, the first of one per invariant. DIAGRAMS. A top-level system diagram in the overview showing authored files through fragments, composition, resolution and both delivery classes back to the cluster -- and resolved.yml returning to the owning repository. A sequence diagram in chapter 50 tracing one change end to end across six actors. Eleven mermaid blocks now, across flowchart, classDiagram and sequenceDiagram. The validator was extended to check sequence participants as well as flowchart endpoints, and to fail on a diagram type it does not know rather than skipping it silently. Audit clean: 11 diagrams valid, 18 example files parsing, no broken links, all four workflows single-job.
ExtraToast
changed the base branch from
docs/ch40-composition
to
v1-pre-release
August 31, 2026 13:42
ExtraToast
changed the base branch from
v1-pre-release
to
docs/ch40-composition
August 31, 2026 13:42
ExtraToast
added a commit
that referenced
this pull request
Aug 31, 2026
Lands the remainder of the v1 specification. Chapters 10 and 16 merged as #58 and #59; this carries 20, 30, 40, 50 and 60 together, plus the revisions those chapters make to earlier ADRs and the examples. Combined rather than stacked, because the stack could not be salvaged after the first two squash-merges: each branch was cut from the pre-squash tip of the one below, so rebasing them onto the merged base conflicted. Splitting them again after the fact would also have produced PRs that do not stand alone -- chapters 50 and 60 revise 30 and 40 (prune semantics, class C's home, two resolved open items), and the glossary correction touches all of them. #60 through #63 remain as the per-chapter review record. Chapter 20, the Resolved Deployment. One rule carries the specification: every assignment is a pure function of Service Intent, the pinned Cluster Context and the pinned locks. Nothing is allocated or remembered between renders. Applying it exposed a hole in ADR-0003 -- taken literally it forbids declaring the Service Id -- so identity is separated from pool: identity is declared and checked, a pool is assigned. Not one live hostname derives from a Service Id, so an exposure carries a name. Also corrects something said repeatedly and wrongly: layer 2 assigns a selector and an affinity, never a node. Chapter 30, the Deliverable Set. Closes the coverage question with a measurement: 450 objects, 39 kinds, 329 files, split 364 intent-derived, 41 pack-delivered, 45 authored. 328 of the 364 have an adapter; the 36-object gap is 14 that need only registering and 22 that need writing. Chapter 40, composition. 26 estate-wide invariants across identity, references, secrets and completeness. The unit of publication is a repository rather than a domain, which resolves the homelab-collections split question without a decision. Order-independence forces every collision to be an error rather than a last-write-wins merge. Chapter 50, lifecycle, with ADR-0019. Push delivery gated by relationship-scoped aggregators; Flux keeps the foundation, because 18 of the 41 pack-delivered objects are HelmRelease and applying one with kubectl does nothing without helm-controller. The cluster is the inventory: a deployer label supplies what Flux gets from a Kustomization's record. Drift correction is an in-cluster CronJob, not an Actions schedule, because the estate measured Actions crons at four to seven runs per day regardless of the declared interval. Chapter 60, setup and adoption. The ordering hazard is the point: Flux prunes, so spec.prune must be set false BEFORE a Service's manifests leave the Flux source, or adoption deletes it. Also: the glossary was materially stale and is corrected -- Claim Mode described a model that no longer exists, Contended Value conflated identity with pool, and seven terms were missing. 39 terms to 47. Four workflows, an Aggregator declaration, a Renovate custom manager, the generated deployer RBAC, the reapply CronJob and a negative fixture. Every workflow is one job with many steps, because jobs bill by the minute rounded up. Audit at commit time: 11 mermaid blocks valid across three diagram types, 18 example files parsing, no broken relative links, no stale vocabulary.
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Chapter 50, plus ADR-0019 and revisions to ADR-0006, ADR-0014, ADR-0015 and chapters 30 and 40. Stacked on #62 → #61 → #60 → #59 → #58.
Replaces pull-based GitOps delivery with a push-based one gated by relationship-scoped aggregators.
The boundary was already measured
Chapter 30 split coverage into 364 intent-derived / 41 pack-delivered / 45 authored objects for entirely unrelated reasons. Delivery splits on the same line — and class B cannot move, for a concrete reason rather than an architectural one:
18 of those 41 objects are
HelmRelease, and applying one withkubectlaccomplishes literally nothing without Flux'shelm-controller. Those charts arevault,vault-secrets-operator,metrics-stack,traefik×2,cert-manager,external-dns,metallband all observability — the Secret Store, the mechanism that delivers every secret under ADR-0005, Prometheus, every route, TLS, DNS and LoadBalancer addresses.Rendering them to plain manifests was considered and rejected: it means owning 18 upstream charts' values, hooks and CRD upgrade paths, and
vaultplus VSO are in the set — so a bad render breaks secret delivery for everything.Resolves chapter 30's class C
The 45 homeless Grafana objects divide by nature: 14 service-specific dashboards become Assets on the owning Service, 3 runtime-family dashboards (
spring-boot-jvm,spring-boot-2.1,spring-service-endpoints) ship with the Runtime Profile alongside theOTEL_*/PYROSCOPE_*values it already injects, 14 platform dashboards ship in the observability pack, andservice-overview/service-templatebecome derived per Service fromscrape,runtimeandexposure—service-templatebecomes the renderer's template rather than a cluster object, which is what its name always implied.Coverage can now reach 100%, which matters because the ledger's own header says "every entry is a deferred fix, not a permanent exemption."
What is given up, stated plainly
Pruning. Flux prunes because a Kustomization keeps an inventory;
kubectlkeeps none. So every applied object carries adeployerlabel and the delete pass is the difference between that query and the render. The cluster is the inventory — nothing to lose, nothing to resync, and no first-run special case: on adoption nothing is labelled, so the delete pass deletes nothing and adoption leaves orphans for the coverage assertion instead.Continuous reconciliation. Replaced by an in-cluster CronJob, not a GitHub Actions schedule —
CLAUDE.mdmeasures why: "four to seven runs per repo per day regardless of the declared interval… Never build anything needing prompt reaction on a schedule alone, and do not raise the frequency to compensate." Drift correction is exactly such a reaction. A cluster CronJob is also the patternvault-metrics-token-renewalalready uses.A single answer to what is live. Given up deliberately: forcing convergence would let one red aggregator halt the estate, which is the coupling aggregators exist to break. Lag is measured instead, and cross-slice removals go through expand/contract — checkable by composition, since it sees both sides of every inbound derivation.
E_CONTRACT_TOO_EARLYcovers theauth-apiCORS case: removing an origin while it is still deployed at an older lock breaks production.One property improves. Server-side apply reports a field-ownership conflict when a human edited a field the aggregator owns, rather than silently reverting it as reconciliation would.
Enforcement, not convention
The
deployslist generates the aggregator's ServiceAccount and Role, so "deploy authority is exactly one" is enforced by the API server — a workflow applying a Service it does not own gets a 403. That makes therbacadapter pay for itself twice: it was already 16 of the 36 objects in chapter 30's coverage gap.Upstream changes
SystemTestProjectbecomes the deploy unit, gainingdeploysalongsideexercises.lock.spec.dependencyGraph.order, whichapply-candidate.mjsalready does.Most of the test stage exists
deploy-harnessalready hasprovision-vcluster.mjs,apply-candidate.mjs(server-side apply per DAG layer, route rewriting), the SC-6 triple-digest assertion, scoped Vault AppRoles per test namespace, CrashLoop and VSO verification, sharded execution with retry and transient classification, an owner-approved quarantine registry and guaranteed teardown. It is wired to one central compose gate rather than to N aggregators — the work is rewiring, not writing.Four open items
Aggregator CI cost (~6 suites per change, against
CLAUDE.md's measured billing shape of "561 minutes billed 2,845"); whether the test vcluster keeps Flux for class B; the lag bound's actual N; and what narrowsdeploy/productionto the foundation, which ADR-0008 in the workspace currently describes.