Proposal: first-class `fbuild cache save / restore` for toolchains + platforms + framework + zccache sidecar

[zackees]

Background

actions/cache@v4 is what every consumer of fbuild reaches for to keep ~/.fbuild/prod/cache/ warm across CI runs (example). It works, but it is suboptimal for fbuild specifically:

• Serial untar. Default tar extract is single-threaded. On Windows runners every newly-extracted file fires a Defender on-access scan, so a 1.1 GB ARM GCC toolchain extract that takes ~30 s on Linux can take 3–5 min on Windows.
• One blob, no shape. actions/cache@v4 has no notion of "just the toolchains" vs "also the zccache core/" vs "also the framework checkouts" — consumers either cache the whole ~/.fbuild dir (correct but slow) or hand-pick paths in YAML (fragile, drifts when fbuild moves things).
• Caller has to know fbuild internals. Consumers should not have to know that index.sqlite is co-required with installed/ or that core/ is owned by zccache. That is fbuild internals leaking into every workflow.
• zccache cache is its own thing. zccache already has zccache gha-cache save/restore (tar+gzip, integrates with @actions/cache API). Right now core/ either gets caught up in the consumer's actions/cache@v4 blob (and uses zccache's primitive nowhere) or gets missed entirely.

Proposal

Add first-class fbuild cache save / restore subcommands that own the archive format, the layout, and the zccache sidecar. The shape is deliberately close to soldr's cache_lib::save / save_delta / load (soldr/crates/soldr-cli/src/cache_lib/save.rs), which has already solved most of the same problems for Rust target caches in setup-soldr.

CLI

fbuild cache save    <archive.tar.zst> [options]
fbuild cache restore <archive.tar.zst> [options]
fbuild cache list    <archive.tar.zst>           # manifest dump, no extraction
fbuild cache verify  <archive.tar.zst>           # manifest hash + index.sqlite integrity

Default save = everything fbuild owns (toolchains, platforms, framework checkouts, downloaded archives, sqlite index, installed marker) plus delegates core/ to zccache via a sidecar invocation. Restore is the inverse.

Options

Option	Default	Notes
--include <slice,...>	all	toolchains, platforms, archives, installed, index, zccache. Leave room for future library-selection, avr8js-node.
--exclude <slice,...>	none	Subtract from --include.
--zstd-level <N>	9	Matches setup-soldr's measured default: 9 benches at ~12 s on a 140 MB delta vs ~104 s at level 19 (zccache CI v0.9.35, zccache#310). Range 1..=22.
--threads <N>	num_cpus	Sizes the rayon pool used for the parallel walk + parallel extract. Mirrors soldr's SOLDR_LOAD_WORKERS.
--zccache-sidecar / --no-zccache-sidecar	--zccache-sidecar when zccache is on PATH	Save invokes zccache gha-cache save for the core/ slot; restore invokes zccache gha-cache restore. When off or zccache absent, core/ is treated as a vanilla fbuild slice (slower, but works).
--cache-dir <PATH>	~/.fbuild/prod/cache	Override fbuild's cache root (mostly for tests and the dev/ profile).
--sketch / --no-sketch	--no-sketch	Sketch build artifacts (<project>/.fbuild/build/) are NOT cached by default. The flag exists so power users can opt in for monorepo caching scenarios, but the headline use case (cross-CI-run cache warming) gains nothing from sketch caching.
--verbose	off	Per-slice byte / file counts, ratio, parallel-worker stats.

Archive format

Same shape as soldr's .tar.zst:

FBUILD_MANIFEST.pb               # protobuf manifest (version, slices, byte counts, hashes)
toolchains/...                   # if included
platforms/...                    # if included
archives/...                     # if included
installed/...                    # if included
index.sqlite                     # if included
zccache.tar.zst                  # nested archive produced by zccache (or equivalent)

The manifest carries:

• Slice inventory (which slices were saved + byte counts + content hashes)
• fbuild version + cache-layout version (so a v0 archive does not try to restore into a v1 cache shape and silently rot)
• A zccache_sidecar { format = "gha-cache-tar-gz", level = N } block when the sidecar fired

Manifest is hand-written prost types (no protoc build dep), same trick soldr is already using (cache_lib/save.rs).

Implementation notes (cribbing from soldr)

• Parallel walk. jwalk::WalkDir with Parallelism::RayonNewPool(threads). On the 1.1 GB ARM toolchain, the walk is ~50x faster on Windows vs serial std::fs::read_dir because Defender fires one callback per file regardless of walker.
• Parallel restore. tar deserialization is inherently serial, but file writes are not. Soldr's load() streams the tar into a bounded mpsc::sync_channel, then a rayon pool drains it. Same model here. FBUILD_RESTORE_WORKERS env override for ops teams who want to tune.
• zstd-9 default. Explicitly justified in setup-soldr's action.yml: 9 = ~12 s on a 140 MB delta vs ~104 s at 19. Do not ship lower than 9 (compression ratio matters for free GHA cache quota); do not ship higher than 9 (diminishing returns get severe past 12).
• mtime snapshotting? Soldr does this because Cargo fingerprints mtimes. fbuild's compile cache (zccache) is content-hash addressed; the framework/toolchain dirs are write-once-read-many. → No mtime snapshotting needed in v1. Document this here so it is an explicit decision, not an oversight.

Slice decisions (with rationale)

Slice	Default in save?	Rationale
toolchains/	yes	1.1 GB+. The whole point.
platforms/	yes	Framework checkouts (framework-arduino-lpc8xx, etc.) — git clones we want to skip.
archives/	yes	Pre-extracted tarballs from developer.arm.com etc. Re-downloading is the slowest possible action.
installed/ + index.sqlite	yes	Required for fbuild to know what is already installed. Tiny, no reason to exclude.
core/ (zccache)	yes — via sidecar	Per the user-asked design question: yes, sidecar by default. Compile cache makes warm runs 10x faster. Delegated to zccache so we do not reinvent its content-addressed format.
library-selection/	optional (not v1)	Small, per-project — not worth the format-version churn now.
avr8js-node/	optional (not v1)	Only used by the AVR emulator path; orthogonal.
worktrees/	never	Per-checkout state; caching across runs is wrong by construction.
<project>/.fbuild/build/ (the sketch)	off by default	Per user direction. Output is derived from sketch source which changes every run; caching it inverts the win. Flag exists for monorepo scenarios; not on by default.

Why not just let zccache handle it

Considered. The reason zccache owning everything does not work:

• zccache does not know about fbuild's index.sqlite schema and should not have to. The "what packages are installed" state is fbuild's by definition.
• zccache's gha-cache save is actions/cache-backed — it requires the GHA cache API env vars (ACTIONS_CACHE_URL + ACTIONS_RUNTIME_TOKEN). fbuild's archive format needs to also work for cold runs, file:// URLs, local dev, CI providers that are not GitHub, etc.
• The "let zccache handle everything" approach pays a perf price on the toolchain slice because zccache's per-file content-addressed format is great for .o files (where dedup wins) and worse for 50,000-file unpacked toolchain trees (where a single tar streams faster).

So: fbuild owns the framework/package/library/dependency layer; zccache owns the compile-cache layer; the sidecar in fbuild cache save glues them so the user sees one archive on disk.

Consumer-side migration path

Today (workflow YAML):

- uses: actions/cache@v4
  with:
    path: |
      ~/.fbuild/prod/cache/toolchains
      ~/.fbuild/prod/cache/platforms
      ~/.fbuild/prod/cache/archives
      ~/.fbuild/prod/cache/core
      ~/.fbuild/prod/cache/installed
      ~/.fbuild/prod/cache/index.sqlite
    key: fbuild-${{ runner.os }}-${{ inputs.platform }}-${{ hashFiles('platformio.ini') }}

After (workflow YAML):

- run: fbuild cache restore ~/fbuild-cache.tar.zst || true
- run: fbuild build . -e ${{ inputs.platform }}
- run: fbuild cache save ~/fbuild-cache.tar.zst
- uses: actions/cache@v4
  with:
    path: ~/fbuild-cache.tar.zst
    key: fbuild-${{ runner.os }}-${{ inputs.platform }}-${{ hashFiles('platformio.ini') }}

Net wins for consumers:

• One cached path, not six.
• They stop having to know which subdirs of ~/.fbuild/prod/cache/ exist.
• Restoring on Windows runners stops being throttled by Defender.

v1 scope (proposed)

• fbuild cache save <archive> with --include, --exclude, --zstd-level, --threads, --zccache-sidecar flags
• fbuild cache restore <archive> with --threads flag
• fbuild cache list <archive> and fbuild cache verify <archive> (cheap, manifest-only)
• Hand-written prost manifest types (no protoc build dep)
• Default level = 9, default include = all, default threads = num_cpus
• Test suite: round-trip on each slice combo; smoke test against a real fbuild cache dir on Linux + Windows runners

Out-of-scope (defer to v2)

• Delta archives (soldr does this with save_delta; cheap once v1 lands but not the v1 motivator)
• mtime snapshot/replay (not needed for fbuild's cache shape)
• Sketch caching (<project>/.fbuild/build/) — flag-gated for power users
• library-selection/ slice + avr8js-node/ slice — extra slices are version bumps; queue them for v2 once the format is stable
• Cross-host portability of the toolchain slice (toolchains are arch-specific by definition; an ubuntu-latest archive cannot restore on windows-latest anyway)

Why now

zackees/ArduinoCore-LPC8xx#17 just shipped a "cache the right six paths" workaround for the same project. Every downstream consumer is going to grow the same workaround. Centralizing this once in fbuild prevents the workaround from being copy-pasted into every project's CI.