⚡ speed up CI and deploys: registry layer cache, credential-gated consumer roll, direct-change version stamps - #677
Merged
Merged
Conversation
The 10GB Actions cache quota evicted buildx layer blobs on almost every run, so the k3d smoke and the publish jobs rebuilt every image cold. The layer cache now lives in one ghcr repository (opencrane-buildcache) with a tag per deployable, the smoke exports its layers for the next run, and the npm download cache is shared between Dockerfile stages.
Every deploy restarted opencrane-server, litellm, and mcp-gateway right after the helm upgrade that had just rolled them, forcing a second full startup of the heaviest workloads. The finalization now stamps a checksum of the published connection Secrets onto the pod templates: an unchanged checksum is a server-side no-op, a changed one triggers exactly one rollout, and a run that failed between publish and roll self-heals on the next deploy.
A shared library, root dependency, or lockfile change forced every application to restate the root version, so most CI failures were manifest bookkeeping instead of real defects. The release gate now requires the root-version stamp only from applications whose own files changed; untouched applications keep their latest released version, which images pinned by commit SHA already made safe. The 0.9.2 manifest moves with the policy: artifact-service and channel-proxy stamp to 0.9.2 for their direct changes since 0.9.1, while agent-controller, artifact-preprocessor, and artifact-scanner return to 0.9.1 because only propagation had bumped them.
The Chromium install and Storybook chain ran serially inside the test job and its apt phase has hung an untimed job for a long stretch, holding a runner and starving the queue. Storybook visual regressions now run as their own parallel job with cached browser binaries (the apt install only happens on a cold cache and is capped at ten minutes), every job carries a timeout, ripgrep installs as a pinned static binary instead of a full apt round-trip, and the layer cache splits into a trusted and a pull-request repository so unreviewed layers can never reach a published image.
8 tasks
📝 document CI, deploys, warnings, and version migrations in docs/ci-and-deploy.md
…I-managed deployment Five pages under website/contributing ground the new docs/ci-and-deploy.md reference for site readers, wired into the nav and sidebar; the website stamps to 0.9.2 for its direct change.
The PostgreSQL-bound work (migration convergence, generated client, target baseline, SQL authority suites) and the API-contract server rebuild ran serially inside the test job. Each now runs as its own parallel job gating publication, taking minutes off the test job's critical path.
Collaborator
Author
|
The k3d smoke failure on this PR is inherited from develop, not introduced here: every run since #673 merged fails the new public-health completeness gate with |
…edups # Conflicts: # apps/_infra/deploy-k8s/charts/opencrane-channel-proxy-0.8.0.tgz # apps/channel-proxy/helm/Chart.yaml # releases/0.9.2.json
Both the old unconditional restart and the checksum roll forced a second boot of the heaviest workloads on fresh installs, where nothing needs propagating: the pods start after this run published the Secrets. Since the public health report and initial model seeding grew the server's boot, that second roll stopped converging inside the blanket 300s wait on the loaded smoke runner. Fresh installs now skip the roll, and the smoke gets 600s of headroom that a healthy wait never uses.
Seeding a placeholder model provider (added while chasing the models probe) made the server fetch a BYOK Secret through the API server and exit fatally when that call failed on the loaded runner, turning one unavailable probe into a CrashLoopBackOff. CI holds no provider credentials, so the smoke stops pretending otherwise: it asserts the report is complete, every service it can provision is healthy, and model routing may be unavailable. A broken channel, memory, file, or database probe still fails the gate.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
CI and deploys are slow, and measurement showed where the time goes:
build-and-pushrebuilt the same images again. The smoke never wrote cache, so PR iteration always built cold.rollout restartright after the helm upgrade, purely to propagate connection-Secret changes.What
type=ghato ghcr (opencrane-buildcache+opencrane-buildcache-pr). Publishable builds read only the trusted cache (written by integration pushes); same-repo PR builds read both and write only the PR cache, so unreviewed layers can never reach a published image. Fork PRs read trusted, write nothing. This also frees ~8GB of Actions cache quota for the node/Nx caches being evicted alongside.timeout-minutes.check:release-versioningrequires the root-version stamp only from applications whose own files changed; untouched applications keep their latest released version (images are SHA-pinned, so shared changes reach them regardless). The 0.9.2 manifest moves with the policy: artifact-service and channel-proxy stamp up; agent-controller, artifact-preprocessor, and artifact-scanner return to 0.9.1 (only propagation had bumped them).keys/folder) and a five-page Contributing section on the website, wired into nav/sidebar, site build validated.Remaining caching/split opportunities are tracked in #678.
Validation
check:release-versioning --base origin/develop→ PASS;test:release-versioning49/49;test:affected-deployables15/15run-helm-contracts.sh→ all contracts PASS (database-migration contract updated for the checksum roll)check:module-growth,config-docs-coverage --strict,agent-style-check→ clean; workflow YAML + shell syntax validatedbuild→ success, no dead linksExpected effect: warm smoke ~5–6 min instead of 13–18,
build-and-pushnear cache-hit, no double rollout on deploys, no more hung-apt runner starvation, and shared-change PRs stop failing on untouched packages' versions.