docs: chapter 20, the Resolved Deployment - #60
Closed
ExtraToast wants to merge 6 commits into
Closed
Conversation
Defines layer 1 normatively: the Service root, the Workload, and every field grouped by concern, each justified against one of the two rules -- no mechanisms, and no contended values. Adopts `intent.jorisjonkers.dev/v1` for the document namespace rather than reusing `deployment.jorisjonkers.dev`, which three mutually incompatible documents already share. Includes an explicit forbidden list: eighteen things that are a build error in layer 1, each with where the value comes from instead. Three worked examples, all rendered from live state rather than invented: - knowledge: two workloads on two runtimes, five path rules with mixed audiences, an SSH key at 0400 that forced mode: file into the model, an irreplaceable volume, and a deliberate mandatory: false for a consumer with no listener - auth-api: mode: fetch with dynamic credentials and tolerates: reload -- which it already runs in production via SPRING_CONFIG_IMPORT: vault:// and VAULT_DB_ENABLED -- a Transit operation grant, four dependencies, and nine CORS origins that are the inbound edge set maintained by hand - platform-postgres: a third-party image with runtime: none, TCP probes, a static asset kept out of a derived image deliberately, provides consumed by eight services, and an init script that stops being authored because it creates one database per consumer the graph already knows Two shape completions the live cluster forced: probe kinds, because postgres probes tcpSocket and not HTTP; and `lifecycle` replacing `kind`, because `statefulset` is a mechanism. One unification: overriding a Runtime Profile value uses the workload-level `overrides` list only. An earlier illustration also allowed an `overrides:` key inside a config entry, which would be two mechanisms for one thing. One refinement: `dependsOn` sits on the Workload, not the Service, so network policy is precise -- within knowledge, the api reaches Postgres and the ingest worker reaches RabbitMQ without inheriting each other's egress. The service edge set is the union, and that union still drives the reconcile DAG and co-test membership. ADR-0006 and ADR-0011 are unchanged in substance. Diagrams are embedded as fenced mermaid blocks. GitHub does not render a bare .mmd, so a separate file would be invisible in review, and a copy in both places is the duplication this specification exists to remove. Three fields had to be proposed and are marked <<proposed>>: sidecars, size and minAvailable. Listed as open item 6 for grading before approval.
Five revisions, each settled by grilling against live state rather than
assumed, plus one consistency gap they opened.
PORTS. No ports list. A port is an integer written where it is used. The
rendered k8s port name is the name of the provides surface declaring that same
integer, falling back to a role-derived name. That rule reproduces every port
name the cluster already uses, because the live names are surface names
already: http appears 23 times, plus metrics, db, smtp, sieve, s3 and
submissions. containerPort entries are derived -- they are documentational in
k8s, so declaring them would be a third place to state a number.
One thing surfaced and left to settle: live names Postgres's port db while the
natural surface name is postgres. Recorded as a deliberate rename rather than
hidden. Live also contains both submissions and submission, which this rule
happens to expose.
PROBES. readiness and liveness are siblings, each carrying its own path, with
no fallback. The v2 fallback made a specific failure the default: liveness
probing the readiness endpoint turns a dependency outage into a crash-loop,
because readiness means can-I-serve and liveness means am-I-wedged. Two live
workloads rely on that fallback today, app-ui and agents-login. tcp is a first
class alternative because postgres probes tcpSocket on both. probes: none
records a deliberate absence.
CONFIG. Env files, in real dotenv format, per workload -- knowledge-api and
knowledge-ingest-worker share the RabbitMQ coordinates and nothing else.
base.env holds what does not vary, one overlay per cluster holds what differs,
overlay wins key by key. Chosen because dotenv is the one .env format in the
estate that already carries real configuration: stalwart-provisioner's files
hold actual knobs, while six service-repo files are comments only and twenty
collection files are YAML documents whose spec.values holds a namespace. The
base-plus-overlay split also formalises what stalwart already half-invented,
where runtime.env holds invariant values and production.env and staging.env
are byte-identical.
Derived values are named placeholders, not literals: ${dependency:...} and
${secret:...}. This is the substitution mechanism ADR-0007 already defines for
assets, not a new one, and it stays restricted to named placeholders with
declared sources. The placeholder names the source and the key names the
variable, which is what lets knowledge write DB_HOST and n8n write
DB_POSTGRESDB_HOST from the same Postgres.
SECRETS. A separate SecretAccess document owns access and file placement, not
env var names. Access is an intent tier -- read, self-renew, self-roll,
custody -- from which the platform derives the Vault policy. The tiers exist
because metrics-token-renewal.yaml already distinguishes them in production:
renewal needs no privilege and leaves the value unchanged so nothing
downstream re-reads, while minting is the only reason that job has a Vault
identity at all. self-roll derives patch rather than update for the reason the
same file records, so a roller gains no read access to sibling keys -- in that
document, the Discord webhook and the Grafana client secret. custody exists
because agents-api creates and deletes secrets at runtime under a prefix whose
paths cannot be enumerated.
FETCH AND ROTATION, the gap the split opened. delivery is env, file or self.
self is the runtime fetch auth-api already runs through spring-cloud-vault, and
the only delivery achieving zero-downtime rotation. Because delivery is
explicit rather than inferred, a delivery: env grant with no matching
placeholder is a dead grant and fails -- preserving chapter 16's property that
a declaration deriving nothing is ceremony.
The two files check each other, and the check was run against all three
example sets rather than asserted: knowledge 5 placeholders to 5 env-delivered
keys, platform-postgres 1 to 1, auth-api 0 to 0 because all its grants are
delivery: self. No dead grants, no unauthorised references, no delivery: env
paired with tolerates: reload.
ADRs 0005, 0007 and 0008 are revised in place rather than superseded, since all
three are still status: proposed and none has been approved.
Nine example files replace three: a service document, an env file and a
secrets document per service.
Reverses the separate SecretAccess document. A secrets list now sits in
service.yml at whichever level the secret is shared: on the Service when every
Workload holds it, on a Workload when only that one does. A Workload's
effective set is the Service-level list plus its own, with no override or
removal syntax -- a Workload that must not hold a shared secret is evidence the
secret was never shared, and it moves down a level.
The evidence for two levels came from the examples themselves. knowledge holds
six grants across two Workloads and two of them, platform/postgres and
platform/rabbitmq, are identical for both. Declaring those once is the
difference between one edit and two when a key is added.
The access-versus-binding split survives unchanged, because it was never
file-level. Access is declared in service.yml beside the dependsOn edge that
motivates it; the environment variable that carries it is a ${secret:...}
placeholder in the env file. The separate document would only have added a
third file keyed by Workload name -- a join key that can drift -- with no
compensating benefit.
The three example sets now demonstrate all three level patterns: auth-api 3
service-level and 0 workload-level, knowledge 2 and 2, platform-postgres 0 and
1. knowledge gains a second env file so both its Workloads are represented.
The cross-check was re-run against the two-level shape rather than assumed:
knowledge/knowledge-api 5 placeholders / 5 env keys (shared 2, own 1)
knowledge/knowledge-ingest-worker 4 placeholders / 4 env keys (shared 2, own 1)
auth-api/auth-api 0 / 0 (shared 3, own 0)
platform-postgres/postgres 1 / 1 (shared 0, own 1)
No dead grants, no unauthorised references, no delivery: env paired with
tolerates: reload. Two results are worth noting: auth-api reports 0/0 because
all three of its grants are delivery: self, and the ingest worker reports 4/4
despite holding a third grant, because that one is delivery: file and
contributes no environment variable. Both confirm the check does not
false-positive on non-env delivery.
ADR-0005 is revised in place; it is still status: proposed.
Defines what declarations produce, and gives the specification its one machine-checkable property. The dependency edge is three fields. Four things derive from it read outbound: reconcile ordering, dependency coordinates, network policy egress, and co-test membership. A fifth class derives from the same edges read inbound, and those are the derivations a service can never declare locally because no service knows its own consumers: - one database and owning user per consumer, which is all init-databases.sh does in 98 lines - NetworkPolicy ingress, since only the inbound set says who to admit - co-test suite membership, which is why 12 of ~25 test classes exercise auth-api with its consumers rather than alone - browser origin allow-lists, which auth-api hand-maintains as nine hostnames - rotation blast radius, which is the inbound set of a secret path Includes the derivation map: 50 nodes, 65 edges, declared on the left and derived on the right. It is dense deliberately and is not meant to be read by eye -- its value is that three properties are checkable by a script over the attribution table ADR-0016 already requires. Corrects a criterion this specification previously stated wrongly. The overview claimed "any node with two inbound arrows is a bled concern". That is false: a Deployment legitimately draws on image, config, claims, health and placement. Convergence on an object is normal; convergence on the same field of an object is the defect. The three properties are now stated properly: 1. totality -- no deliverable has in-degree zero 2. single authority -- no field has two declaring sites 3. no dead declarations -- no declaration has out-degree zero Property 3 is the one that catches ceremony, and it would have caught two live examples: rollbackTargetRetention, validated and acknowledged by every service and read by no renderer; and platform.layer, which fed a registry and never placement while being wrong in 7 of 7 services. Two worked traces: one exposure declaration producing six artefacts plus retiring the two conformance tests that existed only to detect when those six disagreed; and one credential claim producing nine, the ninth computable only over the composed union. Leaves one question open rather than guessing it. The CORS derivation shape is right but the predicate is not established: an OIDC redirect flow needs no CORS entry, so it is probably inbound edges declaring a browser surface rather than all inbound edges. Recorded in the chapter to confirm before implementing.
The rewrite changed two nodes on the map and one worked trace. config becomes env files (base plus cluster overlay), and gains a second outbound edge: a secret placeholder resolves to an envFrom secretRef rather than a literal env entry, so the env file now feeds both the Deployment and the Secret. claims becomes secrets.yml grants, carrying access and delivery. Its derivations are unchanged in substance -- policy, auth role, secret, network policy, container wiring -- but the declaration site moved out of the Service document. health becomes readiness plus liveness as siblings. The credential trace is rewritten to the new two-file shape and gains the bidirectional check the single-document form could not express: a delivery: env grant with no placeholder is a dead grant, a placeholder with no grant is an unauthorised reference, delivery: env with tolerates: reload is impossible, and self-roll on a path other services read needs acknowledgement. That last check is the one nothing in the estate has today. secret/platform/observability holds the Prometheus token, the Discord webhook and the Grafana client secret, and one CronJob rolls one of those keys. Adds a row to the caught-defects table: a secret granted but never referenced is property 3 again, a declaration with out-degree zero. Revalidated: all three mermaid blocks resolve every edge endpoint, and the map still passes its own property 3 -- every declaration has at least one outbound edge.
Layer 2, and it gets one rule that carries the weight of the whole specification: every assignment is a pure function of Service Intent, the pinned Cluster Context and the pinned locks. Nothing is allocated from a mutable pool, drawn from a counter, or remembered between renders. There is no assignment state, which is why the render hash means anything, why a render is reproducible from its lock, and why publishing assignments back to a service repository cannot drift. The rule has teeth because it forces a decision whenever something cannot be a pure function: such a value moves up into layer 1, where it is declared and checked, or sideways into the pinned Cluster Context, where it is platform data. It may not stay in layer 2 as remembered state. Applying it exposed a real hole in ADR-0003, which this chapter corrects. ADR-0003 says a value is platform-assigned if it must be unique estate-wide OR draws on a shared finite resource. Taken literally that forbids declaring the Service Id, which must be estate-unique -- yet ADR-0004 declares it and checks uniqueness at composition. The rule conflated two things: identity unique, but not from a finite pool -> declared, checked pool drawn from a finite resource -> assigned, or context data Hostnames are identity, and the evidence is decisive: not one live hostname is derivable from a Service Id. knowledge serves kb, auth-api serves auth, home-portal serves the apex, headlamp serves dashboard, gatus serves status, and agents-api serves two -- agents and agents-ws. A derivation rule needing an exception for every case is not a rule. So an exposure entry carries a name, declared and checked unique, and layer 2 assembles the fully-qualified host from name plus tier hostname policy plus cluster domain. That is a pure function, and it removes the only assignment that would otherwise have needed allocation state. Corrects a second thing this specification said repeatedly and wrongly: layer 2 does not assign a node. Kubernetes schedules pods; the platform only constrains where they may land, so layer 2 assigns a selector and an affinity. The case that looks like a node assignment is not one either -- a local-path volume binds to the node holding its PV, and that is existing cluster state, observed rather than decided. The published projection separates observed from assigned for exactly this reason. One model, two views. ResolvedDeployment covers the whole composed estate, because assignments are not separable: hostname uniqueness, the reconcile DAG, inbound-edge derivations and the reader set of a secret path are all global. ResolvedService is a projection of it, filtered to one service and committed back to that repository, so the two cannot disagree about what was decided. Includes the normative assignment catalogue, the determinism and provenance fields reused from artifact-contract, and the distinction from ClusterState -- what should be true versus what is true. Conflating those is how "the Kustomization is Ready" comes to be mistaken for "the consumer sees what you intended". Two open items are added to the overview: the exposure name and apex convention, and the drift check's failure mode.
ExtraToast
added
type: docs
This was referenced Aug 31, 2026
ExtraToast
force-pushed
the
docs/ch16-derivation-map
branch
from
August 31, 2026 13:37
1a4f473 to
7bd8c55
Compare
ExtraToast
changed the base branch from
docs/ch16-derivation-map
to
v1-pre-release
August 31, 2026 13:40
ExtraToast
added a commit
that referenced
this pull request
Aug 31, 2026
Lands the remainder of the v1 specification. Chapters 10 and 16 merged as #58 and #59; this carries 20, 30, 40, 50 and 60 together, plus the revisions those chapters make to earlier ADRs and the examples. Combined rather than stacked, because the stack could not be salvaged after the first two squash-merges: each branch was cut from the pre-squash tip of the one below, so rebasing them onto the merged base conflicted. Splitting them again after the fact would also have produced PRs that do not stand alone -- chapters 50 and 60 revise 30 and 40 (prune semantics, class C's home, two resolved open items), and the glossary correction touches all of them. #60 through #63 remain as the per-chapter review record. Chapter 20, the Resolved Deployment. One rule carries the specification: every assignment is a pure function of Service Intent, the pinned Cluster Context and the pinned locks. Nothing is allocated or remembered between renders. Applying it exposed a hole in ADR-0003 -- taken literally it forbids declaring the Service Id -- so identity is separated from pool: identity is declared and checked, a pool is assigned. Not one live hostname derives from a Service Id, so an exposure carries a name. Also corrects something said repeatedly and wrongly: layer 2 assigns a selector and an affinity, never a node. Chapter 30, the Deliverable Set. Closes the coverage question with a measurement: 450 objects, 39 kinds, 329 files, split 364 intent-derived, 41 pack-delivered, 45 authored. 328 of the 364 have an adapter; the 36-object gap is 14 that need only registering and 22 that need writing. Chapter 40, composition. 26 estate-wide invariants across identity, references, secrets and completeness. The unit of publication is a repository rather than a domain, which resolves the homelab-collections split question without a decision. Order-independence forces every collision to be an error rather than a last-write-wins merge. Chapter 50, lifecycle, with ADR-0019. Push delivery gated by relationship-scoped aggregators; Flux keeps the foundation, because 18 of the 41 pack-delivered objects are HelmRelease and applying one with kubectl does nothing without helm-controller. The cluster is the inventory: a deployer label supplies what Flux gets from a Kustomization's record. Drift correction is an in-cluster CronJob, not an Actions schedule, because the estate measured Actions crons at four to seven runs per day regardless of the declared interval. Chapter 60, setup and adoption. The ordering hazard is the point: Flux prunes, so spec.prune must be set false BEFORE a Service's manifests leave the Flux source, or adoption deletes it. Also: the glossary was materially stale and is corrected -- Claim Mode described a model that no longer exists, Contended Value conflated identity with pool, and seven terms were missing. 39 terms to 47. Four workflows, an Aggregator declaration, a Renovate custom manager, the generated deployer RBAC, the reapply CronJob and a negative fixture. Every workflow is one job with many steps, because jobs bill by the minute rounded up. Audit at commit time: 11 mermaid blocks valid across three diagram types, 18 example files parsing, no broken relative links, no stale vocabulary.
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Chapter 20. Stacked on #59, which is stacked on #58 — review in order; each base retargets automatically as the one below merges.
Layer 2 gets one rule, and it carries the weight of the whole specification:
Nothing is allocated from a mutable pool, drawn from a counter, or remembered between renders. There is no assignment state — which is why the render hash means anything, why a render is reproducible from its lock, and why publishing assignments back to a service repository cannot drift.
The rule has teeth because it forces a decision whenever something cannot be a pure function: that value moves up into layer 1 (declared and checked) or sideways into the pinned Cluster Context (platform data). It may not stay in layer 2 as remembered state.
This PR corrects a hole in ADR-0003
ADR-0003 says a value is platform-assigned if it must be unique estate-wide or draws on a shared finite resource. Taken literally, that forbids declaring the Service Id — which must be estate-unique, and which ADR-0004 declares and checks at composition. The rule conflated two things:
Hostnames are identity, and the evidence is decisive: not one live hostname is derivable from a Service Id.
A derivation rule needing an exception for every case is not a rule. So an
exposureentry carries aname, declared and checked unique, and layer 2 assembles the FQDN fromname + tier.hostnamePolicy + cluster.publicDomain— a pure function. That removes the only assignment that would otherwise have needed allocation state.And corrects something I said repeatedly and wrongly
Layer 2 does not assign a node. Kubernetes schedules pods; the platform only constrains where they may land, so layer 2 assigns a selector and an affinity. The case that looks like a node assignment isn't one either — a
local-pathvolume binds to the node holding its PV, and that is existing cluster state, observed rather than decided. The published projection separatesobserved:fromassigned:for exactly that reason, because re-rendering will not change it.One model, two views
ResolvedDeploymentcovers the whole composed estate, because assignments are not separable — hostname uniqueness, the reconcile DAG, inbound-edge derivations and the reader set of a secret path are all global properties.ResolvedServiceis a projection of it, filtered to one service and committed back to that repository, so the two cannot disagree about what was decided.Also included: the normative assignment catalogue (every row a pure function of stated inputs), the determinism and provenance fields reused from
artifact-contract.schema.json, and the distinction fromClusterState— what should be true versus what is true. Conflating those is how "the Kustomization is Ready" comes to be mistaken for "the consumer sees what you intended".Open items added
exposure[].nameand the apex convention need grading, since the first moves a value the contention test had placed on the platform side. And ADR-0017 requires theresolved.ymldrift check without saying what a stale copy does — block that service's pipeline, or merely report.