sccache: Regenerate local depfiles on direct hits - #2
Conversation
SCCACHE_BASEDIRS can give identical cache keys to builds rooted in different locations. A dependency file backfilled unchanged from a slower cache level can therefore expose the producer's paths on a later direct L0 hit. Use the already validated preprocessor manifest to create a local depfile for the supported GCC -MD form. Fall back to preprocessing for formats that cannot be reproduced safely. This retains direct-mode performance while preventing foreign paths from reaching Ninja. Bump the Huddly package version to 0.17.2.
Review finding
I found no functional defect in the implementation itself. The PR correctly handles Falcon's -MD -MT … -MF … form: it creates a local depfile from the validated direct-mode manifest and filters the cached d The PR is exactly the needed fix, and it is cleanly based on current main; after adding the regression test, I would approve it. |
Problem
With a multilevel
disk,webdavcache andSCCACHE_BASEDIRS, dependency-filebehavior differs between an L1 WebDAV hit and the following L0 disk hit.
The shared entry was produced under
/home/jenkins, while the developer buildruns under
/home/geir. On the WebDAV hit, local preprocessing creates a validlocal
.d, so sccache ignores the cached.d. The raw WebDAV archive is thenbackfilled to disk, including its Jenkins
.d.On the next clean build, the direct/preprocessor manifest hits. No local
preprocessing occurs, and the backfilled
.dis extracted from disk. Ninjathen records
/home/jenkins/.conan2/...dependencies. If/home/jenkinsisnot traversable, a later Ninja invocation fails with
stat(...): Permission denied.Reproduction
Before this change, the second build recorded 4,943 Jenkins paths.
Solution
The preprocessor manifest already contains and validates the exact local
source/header paths associated with the direct cache key. For the supported
Unix GCC dependency form (
-MD, one-MTor-MQ, no-MP), this change:.dfrom those validated manifest paths;.d; andUnsupported dependency modes, path syntax, and platforms fall back to normal
local preprocessing. Behavior without
SCCACHE_BASEDIRSis unchanged.This avoids both unsafe text substitution of producer paths and the cost of
rerunning the compiler preprocessor on every direct hit.
The Huddly package version is bumped from 0.17.1 to 0.17.2.
Verification
Exact empty-L0 to L1-backfill to warm-L0 sequence:
/home/jenkinsis non-traversable.For one real
RoomMap.cppcache entry, the original Jenkins.dand generatedlocal
.deach contained 530 unique dependencies. After normalizing the twoproducer base paths, the sorted dependency sets were byte-identical and had
the same SHA-256 digest.
Timing
Measured on the same Falcon build:
SCCACHE_DIRECT=false, warm L0 averageThe exact non-Icefish command reported by the developer improved from 93.70 s
to 25.99 s with this change (518/518 cache hits).
Tests
cargo fmt --checkcargo test compiler::c::test(17 passed)cargo test compiler::preprocessor_cache::test(11 passed)cargo +nightly clippy --workspace --all-targets --all-features --viapre-commit
git diff --cached --checkThe pre-commit rustfmt hook itself currently invokes
cargo +nightly fmt --with staged filenames, which loses the crate's Rust 2024 edition context and
rejects existing let-chain syntax. The edition-aware
cargo fmt --checkpasses.