harden(ci): enforce the shared-cache contract

[Jesssullivan]

Context

PR #227 established the immediate shared-cache migration on main. Its trusted-main run 29326131530 exposed the remaining cold-local failure: about 30 concurrent actions, six Tectonic actions competing for default_bundle_v33 resources, and three targets timing out under ARC-pod starvation. This follow-up keeps the migration and closes that cold-start gap.

The GF consumer registry records this repository as shared-cache-backed (1,191 cache hits, 0 remote processes). Normal CI must not claim or request executor-backed behavior.

Residual hardening

• enforce explicit cache-read/no-upload behavior on pull requests and cache-write/upload behavior only on trusted main
• validate the upload mode and pass it after Bazel rc expansion
• provision the lockfile-selected Playwright browser before OIDC token minting or SSH key setup, then pass its path explicitly to local Bazel test actions
• run one local Bazel action at a time, cap the Bazel host JVM at 2560 MiB and each Node action at 1024 MiB, serialize local test actions, and stop the Bazel server between warm/check/test/e2e phases inside the fleet-managed ARC runner container
• refresh the short-lived cache credential before warm/check/test/e2e so the 90-minute reliability gate never depends on one token surviving the entire serialized job
• allow 90 minutes for the deliberately serial, reliability-first hosted gate
• restore a 45 MB Tectonic v33 resource cache, warm the three CV targets serially, and expose that cache only to local sandboxes
• reject the runner-local Tectonic cache path when executor-backed mode is explicitly selected
• add Bazel-backed regression coverage for upload, Chromium, Tectonic path, and executor-separation contracts
• remove stale RBE labels from the normal shared-cache workflow, smoke fixtures, and operator docs

Deliberately not included

• no executor-backed promotion or remote-action-execution claim
• no auth, federation, or content-product behavior change
• no reversal of fix(ci): bazel-remote-gates → shared-cache-backed (unbreak all PR gates) #227's rules_tectonic or public TeX-source changes
• no broad HOME or XDG override

Validation

• bash scripts/test-bazel-cache-backed.sh
• bazelisk --output_user_root=/private/tmp/blog-gf-cache-ci-cached-validation test --config=ci-cached --announce_rc //:bazel_cache_backed_contract
  • effective config proved --jobs=1, NODE_OPTIONS=--max-old-space-size=1024, and --local_test_jobs=1
• wrapper regression proves GF_BAZEL_HOST_JVM_MAX_HEAP_MB=2560 becomes the startup option --host_jvm_args=-Xmx2560m and that shutdown addresses the matching server
• bazelisk --output_user_root=/tmp/jess-ghio-bazel-codex-228 test //:bazel_cache_backed_contract //:bazel_graph_hygiene
• npm run test:bazel-graph-hygiene
• clean-main proof: bazel test //static/cv:pdfs_synced_test --config=ci-cached --remote_cache=
• cold Tectonic proof: the first serial three-target build succeeded and populated a 45 MB resource cache
• clean-output-root warm proof: all three CV targets rebuilt successfully against that resource cache without the five-minute relay fetch
• git diff --check
• actionlint parsed the workflow and reported only existing repository warnings

Hosted run 29348014079 completed the serial Tectonic warm, then lost communication during Remote check without a retained source diagnostic. The next head made the failure class observable: run 29350939534 again completed the serial warm, then Kubernetes retained runner pod jesssullivan-blog-dind-94p4h-runner-g8bhj as Failed with runner exit 137 and reason OOMKilled; the dind sidecar completed normally afterward.

Run 29352692909 proved the first bound was too aggressive rather than disproving the approach. Chromium provisioning, cache attachment, the serial Tectonic warm, and every non-Bazel check passed. During Remote check, Bazel reached 5,854 of 5,858 actions while running workspace_package_checks, then its own 1024 MiB heap threw java.lang.OutOfMemoryError: Java heap space; this was not a pod eviction. Run 29357986080 repeated the same Java-heap failure at 1536 MiB and 5,855 of 5,858 actions while live runner RSS peaked at 3552 MiB. The current head raises the Bazel cap to 2560 MiB, keeps one serialized Node action at 1024 MiB, and stops the server between phases so retained analysis state cannot accumulate across the full job.

Run 29359644809 then proved that resource split on the live 2 CPU / 8 GiB ARC pod: serial Tectonic warm, Remote check, Remote test, and Remote e2e all completed successfully, with the Bazel lane finishing in 45m22s. Whole-pod memory peaked at 6640 MiB during Remote check, remained below the pod limit during browser e2e, and dropped below 1 GiB after the final Bazel shutdown while Actions cache compression ran.

Final-head run 29362660354 passed every required check on 7a4cfcd. The initial attachment token and all four phase-local refreshes succeeded; warm, check, test, and e2e then passed, with Playwright recording 330/330 cases on the lockfile-pinned Chromium. This is the merge proof for the credential-refresh head.

Fleet capacity is not owned by this repository and is therefore not encoded as a fixed number in source. A live recheck during the current run found the jesssullivan-blog-dind runner template at 2 CPU / 8 GiB; the repo-level caps remain deliberately lower and portable if the fleet is resized again.

PR #229's unique safeguard is folded into this branch; #229 is superseded.

[Jesssullivan]

Second-layer validation: #227 proved the cache-read mint and tenant attachment, then exposed Tectonic inheriting read-only /root under local shared-cache execution. Commit d0ca10a gives ci-cached actions stable writable /tmp HOME/XDG paths. Local proof: bazel test //static/cv:pdfs_synced_test --config=ci-cached --remote_cache= passed, including all three Tectonic builds.

[Jesssullivan]

Hosted attempt 1 crossed the shared-cache boundary successfully: token mint, cache attachment, Remote check, Tectonic, Vitest, and blog-agent tests all passed. The only red was //:puppeteer_chromium_smoke because /bin/chromium belongs to the GF executor image and does not exist on the shared-cache ARC runner. Commit c452c13 provisions the package-lock-pinned Playwright Chromium before Bazel starts, passes its absolute path explicitly through the wrapper, and adds a contract assertion for that argument. Local wrapper/Bazel contract checks pass; hosted CI is rerunning on this head.

[Jesssullivan]

Coordination update: PR #227 merged into main while this branch's hosted validation was running, so I am not merging this overlapping history as-is.

The completed #228 run still supplied useful evidence:

• shared-cache token mint/attachment, remote check, and remote test passed;
• package-lock-pinned Chromium ran 328/330 e2e assertions;
• the only failures were the two stale public TeX-source expectations in e2e/cv.spec.ts, which fix(ci): bazel-remote-gates → shared-cache-backed (unbreak all PR gates) #227 changed on main.

I am holding this PR until #227's trusted-main cache-write run settles, then will rebase it as a narrow residual hardening change: explicit cache upload policy, contract regression coverage, credential-safe browser provisioning, writable local action paths, and substrate-honest operator docs. No duplicate substrate migration will be merged.