Avoid report artifact name collisions in CI matrix runs

[roark47]

Background

The app-upgrade-agents migration kept CI matrix execution so each case could run independently. In that setup, multiple jobs produced files with the same names, such as report.html, and downstream artifact collection could overwrite or obscure results.

This is a common CI integration problem for any skill-up suite that runs cases in parallel matrix jobs.

Why skill-up should change

skill-up owns the report layout and the official CI integration story. A matrix-friendly output layout should make every case/run artifact uniquely addressable without requiring every user to add custom upload reshaping steps.

What to change

• Support configurable report output naming, for example report.{case_id}.html or an output template.
• Alternatively, provide an official matrix-safe artifact layout and CI component behavior that uploads the full iteration directory under a unique case/run prefix.
• Preserve the current default report.html / report.xml behavior for non-matrix local runs.
• Consider generating an index page when multiple case-scoped HTML reports are collected.

How to verify

• Run two cases as independent CI-style jobs that write to a shared artifact collection location.
• Verify both HTML and JUnit reports remain present and traceable to their case IDs.
• Verify existing local runs still produce the expected default report file names.
• Add tests for report filename templating or matrix-safe output path generation.

Architecture impact

This affects the report workspace layout and CI integration contract. The change should be backward compatible for existing users who expect report.html and report.xml in the iteration directory.

[zpzjzj]

Looking at this against the current implementation, I think most of it is already achievable today — flagging before we add new config surface.

Reports are per-run, not per-case. WriteResults emits a single report.html / report.xml / report.json (plus result.json, benchmark.json/md) into iteration-<N>/, aggregating every case in that run. So report.{case_id}.html doesn't map onto the model — a run isn't scoped to one case, and there's no single case id to template on.

The collision is avoidable with --output-dir. Each matrix job can already write to a distinct location:

strategy:
  matrix:
    case: [case-a, case-b]
steps:
  - run: skill-up run ./evals/eval.yaml --include ${{ matrix.case }} --output-dir results/${{ matrix.case }}
  - uses: actions/upload-artifact@v4
    with:
      name: report-${{ matrix.case }}
      path: results/${{ matrix.case }}/

Each job writes results/<case>/iteration-1/report.html and uploads under a unique artifact name, so nothing overwrites — and every file stays traceable to its case, not just the HTML/JUnit. The identical report.html basename only collides if multiple jobs' files are deliberately merged into one flat directory, which this pattern avoids.

Proposal: document the --output-dir-per-matrix-job pattern as the official matrix-safe layout (and optionally bake it into the CI component) rather than adding report-filename templating. A configurable --report-name would only help the narrow flat-merge case, and would leave result.json / benchmark.* colliding anyway unless we template those too (which diverges from the Anthropic layout).

Leaning toward: close as achievable today + add a docs section. If there's a concrete flat-collection requirement that --output-dir genuinely can't cover, let's capture that use case here first before adding new naming config.