Skip to content

[CI rehearsal] measure hosted-runner costs for the flawd mutation gate - #63

Closed
fohara wants to merge 3 commits into
mainfrom
ci/mutation-rehearsal
Closed

[CI rehearsal] measure hosted-runner costs for the flawd mutation gate#63
fohara wants to merge 3 commits into
mainfrom
ci/mutation-rehearsal

Conversation

@fohara

@fohara fohara commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Measurement probe for the upcoming mutation-testing gate (dogfooding flawd ci, tracked in the flawd repo's dogfood-lash-ci effort). Times four things on ubuntu-latest: flawd install from the public release mirror (with checksum), the unentitled failure mode (expected exit 5), workspace coverage generation with cargo-llvm-cov, and the flawd Dockerfile image build. Not for merge — it will be closed once the numbers are recorded in the real mutation.yml.

@fohara

fohara commented Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

First probe run (33015348586, ubuntu-latest, cold):

Step Time
flawd install from the release mirror (checksum verified) 2s
Unentitled flawd ci refusal exit 5, instant
cargo llvm-cov --workspace incl. toolchain component + cargo-llvm-cov install 114s
cargo test --workspace (separate profile, effectively cold) 151s
docker build -f Dockerfile 164s — not trusted: the Dockerfile masks cargo build failures with `

A second run with a cargo/target cache is queued to measure the warm-runner story. Cold native-toolchain setup lands around 4.5 min, which points at native execution for hosted runners; the docker-vs-native call gets made on entitled comparative runs.

…inel probe

An interrupted 'flawd doctor' run on this checkout left its sentinel
mutation (InclusionRules include_dependencies true->false) live in the
working tree, and it rode into the previous commit unnoticed. The
lash-agent mutant-killing tests caught it in CI immediately — which is
the system working. Filed flawd-side: sentinel probes must be
crash-safe (never mutate the user's tree in place without guaranteed
restore).
@fohara

fohara commented Aug 26, 2026

Copy link
Copy Markdown
Contributor Author

Warm-cache run (33019217068) with an actions/cache cargo+target cache:

Step Cold Warm
cache restore 37s
cargo llvm-cov --workspace 114s 61s
cargo test --workspace 151s 85s
docker image build 164s 157s (no layer cache on hosted runners; also still masked by `

Conclusion recorded in flawd's ci-integration docs: native isolation for hosted runners (~3 min warm setup before mutant execution), Docker for local/self-hosted/compose. Measurements are done — closing this probe PR; the branch stays for re-dispatch until the real mutation workflow lands.

@fohara fohara closed this Aug 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant