Reclaim runner disk space before session/eval-data publish - #922
Draft
AbhitejJohn wants to merge 3 commits into
Draft
Reclaim runner disk space before session/eval-data publish#922AbhitejJohn wants to merge 3 commits into
AbhitejJohn wants to merge 3 commits into
Conversation
The publish-session-data job shallow-clones the multi-GB dashboard-session-data branch and stages the run's session logs, which purge-replay-sessions.ps1 copies through a temp dir (transiently ~2-3x the data). On a valid-token run that produces real trajectory logs, this can exhaust the runner disk (System.IO.IOException: No space left on device), which previously only surfaced when a run happened to draw a working Copilot token and generate real session data. Add a first-step disk reclaim (removing ~20GB of preinstalled tooling the job never uses) plus df -h logging to publish-session-data and publish-eval-data. Neither job runs dotnet build and pwsh 7 lives under /opt/microsoft, so removing /usr/share/dotnet et al. is safe. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Adds a benign 'When to Use' bullet to exercise the full /evaluate pipeline (discover -> evaluate -> publish-session-data -> comment) on PR #922 so the runner disk-space fix, the eyes reaction, and the result comment format can be verified end-to-end on a real runner. To be reverted before merge. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Collaborator
Author
|
/evaluate |
Contributor
📊 Skill Evaluation Results1 skill(s) evaluated — 1 improved, 0 no credible improvement. A skill passes only on a credible improvement over baseline (mean preference > 0 with its 95% CI above 0).
🔍 Full Results - additional metrics and failure investigation steps ▶ Sessions Visualisation -- interactive replay of all evaluation sessions |
Removes the temporary csharp-scripts SKILL.md bullet added only to exercise the /evaluate pipeline on this PR. End-to-end verification is complete (eyes add/remove, result comment, Sessions/Analytics links all confirmed), so #922 is restored to the disk-fix-only change. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
publish-session-dataintermittently fails withSystem.IO.IOException: No space left on device. Root cause is runner disk exhaustion, not a code bug:dashboard-session-databranch ofdotnet/skills-data, whose working tree is ~8.66 GB and growing.purge-replay-sessions.ps1then copies the merged session set through a temp dir (transiently ~2–3× the data).publish-eval-data(scheduled) faces the same risk as its branch grows.Fix
Add a first-step disk reclaim to both
publish-session-dataandpublish-eval-datathat removes ~20 GB of preinstalled tooling neither job uses (nodotnet build; pwsh 7 lives under/opt/microsoft, not/usr/share/dotnet), plusdf -hlogging so any future disk pressure is diagnosable from the log.Validation
actionlint -shellcheck= -pyflakes=onevaluation.yml— clean (exit 0)bash -nsyntax check of the reclaim snippet — clean (exit 0)|| true),df -h /workssudoonubuntu-latest).Follow-ups (not in this PR)
COPILOT_PAT_*org secret in the copilot-pat-pool (masks disk failures and silently empties eval legs).Copy→Moveinpurge-replay-sessions.ps1to halve transient footprint.