From d9e715b80daad7034d1bf65fe9696d8dd7d8d537 Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Wed, 19 Aug 2026 20:36:04 +0200 Subject: [PATCH 01/26] docs: design for shardable run artifacts and a merge command MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Splits a twd-cli run across parallel CI jobs via `--shard i/n`, has each shard write a machine-readable report plus raw Istanbul coverage, and adds `twd-cli merge ` to join them back into one report. In-process Puppeteer parallelism was already tried and was too flaky — separate jobs share no CPU, dev server, or browser. What blocked that was the lack of any joinable output: every structured value runTests() builds is printed and discarded. Key decisions: - A merged report is shape-identical to a single-shard report, so merge is associative and existing formatters work on both. - Round-robin slicing needs no advance knowledge of the test count; each shard enumerates the suite itself, as runs already do. - discovery.fingerprint makes shards prove they saw the same test set, turning conditional test registration from a silent green into an error. - The coverage failure gate moves from shard level to merge level, so a red shard can no longer silently understate merged coverage. - Merge owns the final exit code. Co-Authored-By: Claude Opus 5 (1M context) --- ...26-08-19-shardable-run-artifacts-design.md | 392 ++++++++++++++++++ 1 file changed, 392 insertions(+) create mode 100644 docs/superpowers/specs/2026-08-19-shardable-run-artifacts-design.md diff --git a/docs/superpowers/specs/2026-08-19-shardable-run-artifacts-design.md b/docs/superpowers/specs/2026-08-19-shardable-run-artifacts-design.md new file mode 100644 index 0000000..1b4b896 --- /dev/null +++ b/docs/superpowers/specs/2026-08-19-shardable-run-artifacts-design.md @@ -0,0 +1,392 @@ +# Shardable run artifacts and a merge command - Design + +Date: 2026-08-19 +Status: Ready to plan +Repo: `twd-cli` only. No twd-js changes. + +## Problem + +A twd-cli run is a single Puppeteer process walking the whole suite. On a large +suite that is the slowest step in CI, and there is no way to divide it. + +Running Puppeteer concurrently *inside* one job was tried and abandoned: several +browsers contending for one runner's CPU made runs flaky enough to be useless. +Separate CI jobs avoid that entirely, because each job gets its own runner, its +own dev server, and its own browser. Nothing is shared, so nothing contends. + +What blocks that today is not concurrency. It is that a run has no +machine-readable output. Every structured value `runTests()` builds is printed +and discarded; the function returns a bare boolean (`src/index.js:359`). Four +parallel jobs would produce four consoles and nothing joinable. + +## Scope + +In scope: splitting a run across jobs, writing each job's results to disk, and +merging those results back into one report covering test results, coverage, and +contract validation. + +Out of scope: concatenating per-shard videos (each shard records its own clip; +the report lists them). Timing-aware shard balancing (needs a persisted timing +artifact — revisit if count-based balance proves uneven). Dynamic matrix sizing +and a `twd-cli list` command (a fixed matrix needs neither; see "The shard +model"). Reducing contract-report noise, which is its own tracked concern. + +This spec also does not reopen the AI-friendly-output decision that there is one +console format and no `--reporter` flag +(`docs/superpowers/specs/2026-07-06-ai-friendly-output-design.md`). The report +JSON is an artifact for merging, not an alternative console reporter. Text +output stays the default and is unchanged. + +## What already exists + +The data is already the right shape; only the output layer is missing. + +| Concern | Exists today as | Merge story | +|---|---|---| +| Test results | `testStatus` = `[{id, status, error?, retryAttempt?}]` plus `handlers` (`src/index.js:281`, `:210`) | Concat. Each test runs in exactly one shard, so ids never collide. `handlers` is identical in every shard. | +| Coverage | `.nyc_output/out.json`, raw Istanbul JSON (`src/index.js:338`) | Natively mergeable via `istanbul-lib-coverage`. No new merge logic. | +| Contracts | `validateMocks()` returns `{results, skipped}`, plain objects (`src/contracts.js:109`) | Concat. Occurrence counters are per-process but keyed by `testId`, so no cross-shard collision. | +| Summary | Built by `formatRunComplete`, then discarded | Recomputed from merged test results. | + +Sharding primitives exist too: `orderedTestIds()` in `src/testOrder.js` already +yields a deterministic pre-order id list. + +## The shard model + +`--shard 2/4` means "I am job 2 of 4 parallel jobs". The `4` is the job count, +chosen by the workflow author. **The test count is never needed in advance.** + +Each shard enumerates the suite itself — `src/index.js:125` already reads +`window.__TWD_STATE__.handlers` on every run — then keeps every Nth id: + +```js +// src/shard.js +export function selectShardIds(ids, index, total) { + return ids.filter((_, i) => i % total === index - 1); +} +``` + +``` +120 tests discovered (indices 0..119), identical in all 4 jobs +shard 1/4 -> i % 4 === 0 -> 0, 4, 8, ... (30 tests) +shard 2/4 -> i % 4 === 1 -> 1, 5, 9, ... (30 tests) +shard 3/4 -> i % 4 === 2 -> 2, 6, 10, ... (30 tests) +shard 4/4 -> i % 4 === 3 -> 3, 7, 11, ... (30 tests) +``` + +Round-robin rather than contiguous slicing: it balances better when adjacent +tests have similar cost, at the price of suite locality. Nothing in twd-js +requires a suite to run contiguously, so locality buys nothing here. + +Consequences worth stating, because they are what make a fixed matrix safe: + +- The suite can grow without a YAML edit. 200 tests split across the same 4 + jobs; they just take longer. +- **An empty shard is legal.** 3 tests across 4 shards leaves shard 4 with + nothing. It writes a valid report with `tests: []`. Not an error. + +## Report schema + +The load-bearing decision: **a merged report is shape-identical to a +single-shard report.** `shards` is always an array — one entry for a single run, +N after merging. This makes merge associative, lets every formatter work on +both, and makes a normal run the N=1 case with no second code path. + +```jsonc +{ + "schemaVersion": 1, + "shards": [ + { "index": 2, "total": 4, + "startedAt": "2026-08-19T10:00:00.000Z", + "endedAt": "2026-08-19T10:00:12.345Z", + "durationMs": 12345, + "executed": 30, "notRun": 0, "stoppedEarly": false, + "coverageFile": "coverage.json", + "recording": { "file": "login.mp4", "bytes": 481920 } } + ], + "discovery": { "totalTests": 120, "fingerprint": "sha256:abc123..." }, + "selection": { "mode": "shard", "filters": [] }, + "handlers": [ { "id": "...", "name": "...", "parent": "...", "type": "test" } ], + "tests": [ { "id": "...", "status": "pass", "retryAttempt": 2 } ], + "contracts": { "configured": true, "partial": false, "results": [], "skipped": [] } +} +``` + +`handlers` and `tests` reuse the exact shapes already flowing through +`src/index.js:128` and `:226`, so `buildTestPath`, `formatRunComplete` and +`generateContractMarkdown` need no data massaging. `contracts` is +`validateMocks()`'s return value verbatim plus two flags. + +`selection.mode` is `"full"` or `"shard"`. `selection.filters` holds the `--test` +values. The two compose: filters resolve first, then the filtered list is +sharded. + +Coverage is **referenced, not embedded** — `coverageFile` names a sibling file. +This keeps `run.json` readable by eye and keeps coverage in stock Istanbul +format so `nyc` tooling works on it untouched. A coverage blob is routinely +several megabytes; embedding it would make every report unreadable. + +Duration is ambiguous once jobs run in parallel, so two figures are reported: +wall-clock span (`max(endedAt) - min(startedAt)`), which is what the developer +waited, and total compute (sum of `durationMs`), which is what was paid for. For +a single shard they are equal. Both are **derived from `shards[]` at render +time**, not stored as fields — the per-shard timestamps are sufficient, and +storing derived totals would let them drift out of agreement under merge. + +### `discovery.fingerprint` is the safety net + +The fingerprint is a hash of `{ orderedIds: , +filters: }`. + +Round-robin sharding is correct only if every job enumerates an identical test +set. That silently breaks if the app registers tests conditionally — a feature +flag, a date, `Math.random()` — or if two shard jobs somehow build different +code. Without the fingerprint that manifests as tests quietly never running and +a green build. With it, merge refuses and explains why. This is the part of the +design least safe to drop. + +## Coverage: the gate moves up a level + +Today coverage is gated twice (`src/index.js:325`): + +```js +if (config.coverage && !hasFailures && !selectedIds) { +``` + +Both gates change. + +`!selectedIds` exists so a `--test` filter cannot produce a misleading +project-wide number. A shard slice is not a user filter, so the rule becomes: +**collect coverage unless `selection.filters` is non-empty.** A filtered run +still skips, sharded or not. + +`!hasFailures` is the more interesting one. `hasFailures` is per shard, so +applying it at shard level gives the worst outcome: shards 1, 2 and 4 write +coverage, shard 3 goes red and writes none, and merge emits a report that looks +complete while missing a quarter of the code paths. Silent understatement is +worse than absence. + +The same policy therefore applies one level up: + +- **Shards always write `coverage.json`.** No shard-level failure gate, so a + shard's file is never mysteriously absent. +- **Merge writes `.nyc_output/out.json` only when the merged run is green.** + +Net policy is unchanged — a red run yields no coverage — but it is now keyed on +the true global result. + +Where coverage lands depends on whether reporting is active, and the two paths +are mutually exclusive on purpose: + +- **Without `--report`** (today's normal run): `.nyc_output/out.json`, exactly as + now. The only change is that a failing run writes it too. +- **With `--report` or `--shard`**: `/coverage.json` only. It is + deliberately *not* also written to `.nyc_output/out.json`, because one shard's + partial coverage sitting at the path `nyc` reads by default would masquerade as + the whole run's. Under sharding, `.nyc_output/out.json` is written by `merge` + and by nothing else. + +Since a red run still exits 1, no coverage gate can be fooled by the relaxed +failure gate. + +## maxFailures stays per shard + +Cross-job coordination is impossible without an external store, so each shard +gets the full `maxFailures` budget (default 10) independently. Four shards can +therefore accumulate up to 40 failures before all four bail. This is documented, +not fixed. + +A shard that bails no longer skips contract validation. Today `stoppedEarly` +skips it outright (`src/index.js:296`, `:317`); instead it validates what it +collected and sets `contracts.partial: true`, so merge can report exactly what +is missing rather than silently dropping a quarter of the mocks. The console +report gains a partial banner in place of the current skip message. + +## `twd-cli merge ` + +Discovery globs `/*/run.json`, which is exactly `download-artifact`'s +layout (one directory per artifact name). `/run.json` is also accepted for +the degenerate single-report case. + +Validation runs before anything is combined. All of these are fatal: + +- at least one report found +- all `schemaVersion` equal (otherwise: mismatched twd-cli versions across jobs) +- all `discovery.fingerprint` equal +- all `shards[].total` equal, and `shards[].index` covers `1..total` exactly — + no gaps, no duplicates +- no test id appears in two reports + +Combining is then mechanical. `tests`, `contracts.results` and +`contracts.skipped` concat. `shards` concats sorted by index. `handlers` and +`discovery` come from the first report, already proven identical. +`contracts.partial` is the OR across shards. One cross-check: `sum(executed) + +sum(notRun)` must equal `discovery.totalTests`; a mismatch warns, since it +indicates a shard-math bug rather than user error. + +Output goes three places: + +1. Merged report to `--out` (default `.twd/merged-run.json`). +2. Merged coverage to `.nyc_output/out.json`, only when the merged run is green. +3. Markdown to `contractReportPath` when configured — so the existing PR-comment + step at `.github/actions/run/action.yml:37` keeps working unchanged. + +**Merge owns the exit code**: 1 on any test failure, any `error`-mode contract +violation, or any validation failure above. Otherwise 0. + +``` +--- Run complete --- + Passed: 114 | Failed: 6 | Skipped: 0 + Duration: 38.2s wall (2m14s compute across 4 shards) + + Shards: 1 ✓30 | 2 ✗30 | 3 ✓30 | 4 ✓30 + + Failed tests (6): + × Checkout > applies discount code + Expected 90 but got 100 (at http://localhost:5173/cart) +``` + +## Modules + +| Module | Responsibility | +|---|---| +| `src/shard.js` | `selectShardIds(ids, index, total)` | +| `src/runReport.js` | `buildRunReport({...})` -> plain object. No I/O. | +| `src/reportFiles.js` | Write `run.json` + `coverage.json`; discover and read shard dirs | +| `src/mergeReports.js` | `mergeRunReports([reports])` -> same shape, plus the validation above | +| `src/mergeCoverage.js` | `istanbul-lib-coverage` `CoverageMap.merge()` | + +Existing files: `parseArgs.js` learns the new flags; `index.js` slices ids and +calls `buildRunReport` + `writeRunReport` instead of discarding its locals; +`bin/twd-cli.js` gains the `merge` command; `testSummary.js` gains an optional +`shards` param that prints the per-shard breakdown when there is more than one. + +`formatRunComplete`'s loose-argument signature is kept as-is and called with +fields destructured from the report, rather than being rewritten to take a +report object. Less churn, and the formatter stays dumb. + +## CLI flags + +``` +npx twd-cli run --shard 2/4 # implies --report +npx twd-cli run --report # write artifacts without sharding (N=1) +npx twd-cli run --report-dir

# default .twd/run +npx twd-cli merge

+npx twd-cli merge --out

# default .twd/merged-run.json +``` + +Report writing is off by default for plain runs, so no existing run starts +littering the working tree. `--shard` implies `--report`, since a shard that +writes nothing is useless. + +Each flag accepts both `--flag value` and `--flag=value`, matching the existing +`readValue` helper in `src/parseArgs.js`. + +### One deliberate divergence: `--shard` validates strictly + +`src/parseArgs.js:28` silently ignores a malformed `--record-speed`. `--shard` +must not follow that precedent. `--shard 5/4`, `--shard 0/4` and `--shard abc` +would each silently run zero tests and exit 0 — a green build that tested +nothing. Invalid shard specs error and exit 1. + +## Failure modes + +| Situation | Behavior | +|---|---| +| Shard crashed before upload, artifact missing | error, names the missing index — never a silent 3-of-4 green | +| Shard uploaded but bailed at `maxFailures` | merged report carries its `notRun`, contracts flagged `partial`, exit driven by the real failures | +| Shards saw different test sets | error citing conditional registration or mismatched code | +| Same test id in two shards | error — shard math bug | +| Fewer tests than shards | valid: empty shard writes `tests: []` | +| `--shard 5/4`, `0/4`, `abc` | parse error, exit 1 | +| A shard's `coverage.json` absent | that shard does not contribute; merge states the contributor count | +| `merge` on empty or missing dir | error, exit 1 | +| `--shard` and `--test` together | compose; filters feed the fingerprint so differently-filtered shards cannot merge; coverage skipped | + +## CI shape + +```yaml +jobs: + test: + strategy: + fail-fast: false # or one red shard cancels the rest + matrix: { shard: [1, 2, 3, 4] } + steps: + - ...checkout, npm ci, chrome, dev server... + - run: npx twd-cli run --shard ${{ matrix.shard }}/4 + - uses: actions/upload-artifact + if: always() # a red shard must still upload + with: + name: twd-run-${{ matrix.shard }} + path: .twd/run + + merge: + needs: [test] + if: ${{ !cancelled() }} # runs even though a shard went red + steps: + - uses: actions/checkout@v5 # merge reads twd.config.json + - uses: actions/setup-node@v5 + with: { node-version: 24, cache: npm } + - run: npm ci # merge needs twd-cli installed + - uses: actions/download-artifact + with: { pattern: twd-run-*, path: .twd/shards } + - run: npx twd-cli merge .twd/shards # owns the final exit code +``` + +Three easy-to-miss details, all load-bearing: + +- `fail-fast: false`, or the first red shard cancels its siblings and merge sees + gaps. +- `if: always()` on upload, or a red shard uploads nothing and merge cannot + distinguish "shard failed" from "shard never ran". +- `if: ${{ !cancelled() }}` on merge, or a red shard short-circuits the workflow + and the merged summary — the entire point — is never printed. + +Action versions are written as `@v5` above for readability; the real workflow +SHA-pins them, matching `.github/workflows/e2e.yml`. + +## Dependency change + +`istanbul-lib-coverage` is currently present only transitively, via the +`@vitest/coverage-v8` devDependency. It gets promoted to a real `dependency`. +Small and battle-tested, but it means running `npm run lock:linux` afterwards so +the wasm32-wasi transitive deps stay correct for Linux CI. + +## Testing + +Existing constraints hold: no test may require a real browser or a real ffmpeg +binary, and `vi.mock('fs')` auto-mocks `statSync` to `undefined`. + +New files: `tests/shard.test.js`, `tests/runReport.test.js`, +`tests/reportFiles.test.js`, `tests/mergeReports.test.js`, +`tests/mergeCoverage.test.js`. + +Two of the cases are properties rather than examples, and they are what make the +whole scheme trustworthy: + +- **Partition**: the union of all shards equals the input list, with every id + appearing exactly once. +- **Associativity**: `merge([merge([a, b]), c])` deep-equals `merge([a, b, c])`. + +If both hold, sharding cannot silently lose or double-run a test. + +Alongside them: gap detection, fingerprint mismatch, duplicate ids, empty +shards, coverage counts summing across shards, `--shard` parse validation, and +extensions to `tests/runTests.test.js` asserting the shard slice reaches +`runByIds`, the report is written, and **coverage is written despite failures** +(the gate change). + +Unit tests cannot exercise the real Actions plumbing, so +`.github/workflows/e2e.yml` gains a 2-shard-plus-merge run against +`test-example-app`. That is what would catch a missing `if: always()` or a wrong +artifact path. + +## Value + +Wall-clock CI time divides by the shard count without the flakiness that killed +in-process parallelism, because separate jobs share no CPU, no dev server, and +no browser. + +Secondarily, the run report is the machine-readable output twd-cli has never +had. Merging is its first consumer; agent-driven TDD loops reading results +without parsing console text are an obvious second. From b2c119dbebe906ed09ef52d15baa92d32e38347e Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Wed, 19 Aug 2026 20:44:40 +0200 Subject: [PATCH 02/26] docs: drop --report from the sharding design, record maxFailures rationale MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --report only meant "write a report without sharding", which `--shard 1/1` already expresses. Cutting it removes a flag, the "--shard implies --report" rule, and a code path in parseArgs, for something no consumer needs yet. That cascade also removes selection.mode: with reports only existing under --shard, the field would be the constant "shard". Also records why a per-shard maxFailures budget is acceptable rather than merely unavoidable — with the suite divided N ways each shard hits its own limit fast, so the extra failures cost no noticeable time. Co-Authored-By: Claude Opus 5 (1M context) --- ...26-08-19-shardable-run-artifacts-design.md | 37 ++++++++++++------- 1 file changed, 24 insertions(+), 13 deletions(-) diff --git a/docs/superpowers/specs/2026-08-19-shardable-run-artifacts-design.md b/docs/superpowers/specs/2026-08-19-shardable-run-artifacts-design.md index 1b4b896..b2ca39b 100644 --- a/docs/superpowers/specs/2026-08-19-shardable-run-artifacts-design.md +++ b/docs/superpowers/specs/2026-08-19-shardable-run-artifacts-design.md @@ -105,7 +105,7 @@ both, and makes a normal run the N=1 case with no second code path. "recording": { "file": "login.mp4", "bytes": 481920 } } ], "discovery": { "totalTests": 120, "fingerprint": "sha256:abc123..." }, - "selection": { "mode": "shard", "filters": [] }, + "selection": { "filters": [] }, "handlers": [ { "id": "...", "name": "...", "parent": "...", "type": "test" } ], "tests": [ { "id": "...", "status": "pass", "retryAttempt": 2 } ], "contracts": { "configured": true, "partial": false, "results": [], "skipped": [] } @@ -117,9 +117,9 @@ both, and makes a normal run the N=1 case with no second code path. `generateContractMarkdown` need no data massaging. `contracts` is `validateMocks()`'s return value verbatim plus two flags. -`selection.mode` is `"full"` or `"shard"`. `selection.filters` holds the `--test` -values. The two compose: filters resolve first, then the filtered list is -sharded. +`selection.filters` holds the `--test` values. Filters and shards compose: +filters resolve first, then the filtered list is sharded. There is no companion +`mode` field — a report only exists under `--shard`, so it would be a constant. Coverage is **referenced, not embedded** — `coverageFile` names a sibling file. This keeps `run.json` readable by eye and keeps coverage in stock Istanbul @@ -178,9 +178,9 @@ the true global result. Where coverage lands depends on whether reporting is active, and the two paths are mutually exclusive on purpose: -- **Without `--report`** (today's normal run): `.nyc_output/out.json`, exactly as +- **Without `--shard`** (today's normal run): `.nyc_output/out.json`, exactly as now. The only change is that a failing run writes it too. -- **With `--report` or `--shard`**: `/coverage.json` only. It is +- **With `--shard`**: `/coverage.json` only. It is deliberately *not* also written to `.nyc_output/out.json`, because one shard's partial coverage sitting at the path `nyc` reads by default would masquerade as the whole run's. Under sharding, `.nyc_output/out.json` is written by `merge` @@ -193,8 +193,15 @@ failure gate. Cross-job coordination is impossible without an external store, so each shard gets the full `maxFailures` budget (default 10) independently. Four shards can -therefore accumulate up to 40 failures before all four bail. This is documented, -not fixed. +therefore accumulate up to 40 failures before all four bail. + +This is documented, not fixed, and the reason it is acceptable is that the +budget exists to stop CI burning time on a fundamentally broken app — with the +suite already divided N ways, each shard reaches its own limit fast enough that +the extra wasted time is not noticeable. Dividing the budget instead +(`ceil(maxFailures / total)`) was considered and rejected: a shard stopping at 3 +failures is hard to explain from its own log, and it makes the CLI depend on the +shard count to compute a threshold. A shard that bails no longer skips contract validation. Today `stoppedEarly` skips it outright (`src/index.js:296`, `:317`); instead it validates what it @@ -268,16 +275,20 @@ report object. Less churn, and the formatter stays dumb. ## CLI flags ``` -npx twd-cli run --shard 2/4 # implies --report -npx twd-cli run --report # write artifacts without sharding (N=1) +npx twd-cli run --shard 2/4 # a shard; writes a report +npx twd-cli run --shard 1/1 # the non-sharded case: one shard, one report npx twd-cli run --report-dir

# default .twd/run npx twd-cli merge

npx twd-cli merge --out

# default .twd/merged-run.json ``` -Report writing is off by default for plain runs, so no existing run starts -littering the working tree. `--shard` implies `--report`, since a shard that -writes nothing is useless. +Report writing is driven entirely by `--shard`, so no existing run starts +littering the working tree. There is deliberately no separate `--report` flag: +a shard that writes nothing is useless, so the report is a property of sharding, +and `--shard 1/1` already expresses "one shard, write its report". A dedicated +`--report` would add a second flag, an implication rule to document and test, +and a third code path in `parseArgs` for something no consumer needs yet. It is +a one-line addition later if one turns up. Each flag accepts both `--flag value` and `--flag=value`, matching the existing `readValue` helper in `src/parseArgs.js`. From 26b541234a05d16ba98493e7862ff59f7c46d52d Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Wed, 19 Aug 2026 20:45:54 +0200 Subject: [PATCH 03/26] docs: scope behavior changes to sharded runs only, add beta release plan MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A run without --shard must behave exactly as 1.4.0. The two changes that would have leaked into existing runs — relaxing the coverage failure gate and validating contracts after an early stop — are now gated on sharding, each reducing to today's expression when sharded is false. Adds the release plan: 1.5.0-beta.0 published under the beta dist-tag, which publish.yml already routes for prereleases, so `npm install twd-cli` keeps resolving to 1.4.0. Also adds explicit non-regression tests for the two touched conditionals, since inference is not coverage. Co-Authored-By: Claude Opus 5 (1M context) --- ...26-08-19-shardable-run-artifacts-design.md | 67 ++++++++++++++++--- 1 file changed, 56 insertions(+), 11 deletions(-) diff --git a/docs/superpowers/specs/2026-08-19-shardable-run-artifacts-design.md b/docs/superpowers/specs/2026-08-19-shardable-run-artifacts-design.md index b2ca39b..c51732b 100644 --- a/docs/superpowers/specs/2026-08-19-shardable-run-artifacts-design.md +++ b/docs/superpowers/specs/2026-08-19-shardable-run-artifacts-design.md @@ -145,6 +145,31 @@ code. Without the fingerprint that manifests as tests quietly never running and a green build. With it, merge refuses and explains why. This is the part of the design least safe to drop. +## Hard constraint: no change to non-sharded runs + +A run without `--shard` must behave exactly as 1.4.0 does — same console output, +same files written, same exit code. The feature is strictly additive, and every +behavior change below is gated on sharding being active. This is what makes the +work safe to ship as a beta that existing users can install without reading a +migration note. + +Two changes needed scoping to honor this, and both reduce to one extra term in an +existing conditional: + +| Today (`src/index.js`) | Becomes | +|---|---| +| `config.coverage && !hasFailures && !selectedIds` (`:325`) | `config.coverage && filters.length === 0 && (sharded \|\| !hasFailures)` | +| `!stoppedEarly && config.contracts?.length` (`:296`) | `(sharded \|\| !stoppedEarly) && config.contracts?.length` | + +With `sharded === false` each reduces to today's expression exactly — the +`!selectedIds` guard and `filters.length === 0` are the same predicate, since +`selectedIds` is only set by `--test` on a non-sharded run. + +The remaining additions cannot affect an existing run by construction: new flags +are inert when absent, report writing happens only under `--shard`, `merge` is a +new subcommand, and `formatRunComplete`'s new optional `shards` param changes +output only when more than one shard is present. + ## Coverage: the gate moves up a level Today coverage is gated twice (`src/index.js:325`): @@ -160,7 +185,8 @@ project-wide number. A shard slice is not a user filter, so the rule becomes: **collect coverage unless `selection.filters` is non-empty.** A filtered run still skips, sharded or not. -`!hasFailures` is the more interesting one. `hasFailures` is per shard, so +`!hasFailures` is the more interesting one, and per the constraint above it is +relaxed **only when sharded**. `hasFailures` is per shard, so applying it at shard level gives the worst outcome: shards 1, 2 and 4 write coverage, shard 3 goes red and writes none, and merge emits a report that looks complete while missing a quarter of the code paths. Silent understatement is @@ -172,14 +198,14 @@ The same policy therefore applies one level up: shard's file is never mysteriously absent. - **Merge writes `.nyc_output/out.json` only when the merged run is green.** -Net policy is unchanged — a red run yields no coverage — but it is now keyed on -the true global result. +Net policy is unchanged — a red run yields no coverage — but under sharding it is +keyed on the true global result rather than on one shard's. Where coverage lands depends on whether reporting is active, and the two paths are mutually exclusive on purpose: -- **Without `--shard`** (today's normal run): `.nyc_output/out.json`, exactly as - now. The only change is that a failing run writes it too. +- **Without `--shard`** (today's normal run): `.nyc_output/out.json`, written + exactly as now, including still being skipped on failure. No change at all. - **With `--shard`**: `/coverage.json` only. It is deliberately *not* also written to `.nyc_output/out.json`, because one shard's partial coverage sitting at the path `nyc` reads by default would masquerade as @@ -187,7 +213,7 @@ are mutually exclusive on purpose: and by nothing else. Since a red run still exits 1, no coverage gate can be fooled by the relaxed -failure gate. +failure gate on the sharded path. ## maxFailures stays per shard @@ -203,11 +229,12 @@ the extra wasted time is not noticeable. Dividing the budget instead failures is hard to explain from its own log, and it makes the CLI depend on the shard count to compute a threshold. -A shard that bails no longer skips contract validation. Today `stoppedEarly` -skips it outright (`src/index.js:296`, `:317`); instead it validates what it -collected and sets `contracts.partial: true`, so merge can report exactly what -is missing rather than silently dropping a quarter of the mocks. The console -report gains a partial banner in place of the current skip message. +A bailing **shard** no longer skips contract validation. Today `stoppedEarly` +skips it outright (`src/index.js:296`, `:317`); under sharding it instead +validates what it collected and sets `contracts.partial: true`, so merge can +report exactly what is missing rather than silently dropping a quarter of the +mocks. The console report gains a partial banner in place of the skip message. +A non-sharded early-stopped run keeps skipping validation, as today. ## `twd-cli merge

` @@ -363,6 +390,18 @@ SHA-pins them, matching `.github/workflows/e2e.yml`. Small and battle-tested, but it means running `npm run lock:linux` afterwards so the wasm32-wasi transitive deps stay correct for Linux CI. +## Release + +Ships as a prerelease so it can be exercised against a real suite before it +becomes the default install. `package.json` goes to `1.5.0-beta.0`, and the +GitHub Release is marked as a prerelease — `publish.yml:27` already routes +prereleases to the `beta` dist-tag, so `npm install twd-cli` keeps resolving to +1.4.0 and testers opt in with `npm install twd-cli@beta`. No workflow change. + +Per the repo's release process the version bump is its own commit carrying +`package.json`, the lockfile regenerated with `npm run lock:linux`, and a +hand-written CHANGELOG entry. The `conventional-changelog` script is not used. + ## Testing Existing constraints hold: no test may require a real browser or a real ffmpeg @@ -387,6 +426,12 @@ extensions to `tests/runTests.test.js` asserting the shard slice reaches `runByIds`, the report is written, and **coverage is written despite failures** (the gate change). +The constraint above needs its own explicit coverage, not just inference: tests +asserting that with no `--shard` flag, a failing run still writes **no** +coverage file and an early-stopped run still **skips** contract validation. Those +are the two conditionals that were touched, so they are the two that could +silently regress an existing user. + Unit tests cannot exercise the real Actions plumbing, so `.github/workflows/e2e.yml` gains a 2-shard-plus-merge run against `test-example-app`. That is what would catch a missing `if: always()` or a wrong From 46231e3eb4b8ce299bcdbd85e0a58ba25304380f Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Wed, 19 Aug 2026 20:48:36 +0200 Subject: [PATCH 04/26] docs: add per-shard failed count to the report schema The per-shard breakdown line needs to show which shard went red, and merged tests do not record which shard ran them. Co-Authored-By: Claude Opus 5 (1M context) --- .../specs/2026-08-19-shardable-run-artifacts-design.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/docs/superpowers/specs/2026-08-19-shardable-run-artifacts-design.md b/docs/superpowers/specs/2026-08-19-shardable-run-artifacts-design.md index c51732b..ed0873b 100644 --- a/docs/superpowers/specs/2026-08-19-shardable-run-artifacts-design.md +++ b/docs/superpowers/specs/2026-08-19-shardable-run-artifacts-design.md @@ -100,7 +100,7 @@ both, and makes a normal run the N=1 case with no second code path. "startedAt": "2026-08-19T10:00:00.000Z", "endedAt": "2026-08-19T10:00:12.345Z", "durationMs": 12345, - "executed": 30, "notRun": 0, "stoppedEarly": false, + "executed": 30, "notRun": 0, "failed": 0, "stoppedEarly": false, "coverageFile": "coverage.json", "recording": { "file": "login.mp4", "bytes": 481920 } } ], @@ -117,6 +117,11 @@ both, and makes a normal run the N=1 case with no second code path. `generateContractMarkdown` need no data massaging. `contracts` is `validateMocks()`'s return value verbatim plus two flags. +Each shard descriptor carries its own `failed` count. Merged `tests` do not +record which shard ran them, so without it the per-shard breakdown line +(`Shards: 1 ✓30 | 2 ✗30 | ...`) could not be rendered — and knowing *which* +shard went red is most of that line's value. + `selection.filters` holds the `--test` values. Filters and shards compose: filters resolve first, then the filtered list is sharded. There is no companion `mode` field — a report only exists under `--shard`, so it would be a constant. From 44d0182e33dad935a689404bf0f9475000ac72a5 Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Wed, 19 Aug 2026 21:11:25 +0200 Subject: [PATCH 05/26] docs: implementation plan for shardable run artifacts 11 TDD tasks, 78 steps, covering src/shard.js, src/runReport.js, src/reportFiles.js, src/mergeCoverage.js, src/mergeReports.js and src/mergeCommand.js, plus the index.js wiring, the merge subcommand, an end-to-end sharded CI job, and the 1.5.0-beta.0 bump. Two structural decisions the plan locks in beyond the spec: - mergeCommand.js is split from mergeReports.js so the merge stays pure and associative. Completeness checking cannot live inside the merge: a 2-of-3 merge is a legal intermediate value, so rejecting it there would make merge(merge(a,b),c) throw and destroy the property that proves no test is lost or doubled. - Each shard's contract markdown is suppressed. Under sharding every shard would overwrite the others with a fraction of the picture, so merge writes it. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-19-shardable-run-artifacts.md | 2845 +++++++++++++++++ 1 file changed, 2845 insertions(+) create mode 100644 docs/superpowers/plans/2026-08-19-shardable-run-artifacts.md diff --git a/docs/superpowers/plans/2026-08-19-shardable-run-artifacts.md b/docs/superpowers/plans/2026-08-19-shardable-run-artifacts.md new file mode 100644 index 0000000..2a42ec8 --- /dev/null +++ b/docs/superpowers/plans/2026-08-19-shardable-run-artifacts.md @@ -0,0 +1,2845 @@ +# Shardable Run Artifacts Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Let a twd-cli run be split across parallel CI jobs with `--shard i/n`, each writing a machine-readable report plus raw coverage, and add `twd-cli merge ` to join them into one report covering tests, coverage, and contract validation. + +**Architecture:** Each shard boots its own browser, enumerates the whole suite as runs already do, and keeps every Nth test id (round-robin). It writes `run.json` + `coverage.json` to a report dir that CI uploads as an artifact. `merge` reads the downloaded shard dirs, validates they agree, concatenates them into a report of the **same shape**, and owns the exit code. Merged-equals-single shape makes merge associative and lets existing formatters render both. + +**Tech Stack:** Node ESM, vitest, Puppeteer (already present), `istanbul-lib-coverage` (new runtime dependency). + +**Spec:** `docs/superpowers/specs/2026-08-19-shardable-run-artifacts-design.md` + +## Global Constraints + +- **Strictly additive.** A run without `--shard` must behave exactly as 1.4.0 does: same console output, same files written, same exit code. Every behavior change is gated on `sharded` being true. +- Work happens on branch `feat/shardable-run-artifacts` (already checked out). Never commit to `main`. +- ESM only. `import`, no `require`. +- No test may require a real browser or a real ffmpeg binary. `node:child_process` and `page.screencast` stay mocked. +- `vi.mock('fs')` auto-mocks `fs.statSync` to return `undefined`. Anything reading a `Stats` must tolerate that. +- One `src/` module per responsibility, one `tests/.test.js` per module — the existing repo convention. +- After any dependency change run `npm run lock:linux` (Docker must be running). macOS npm never installs the wasm32-wasi optional packages, so it leaves their `@emnapi/*` transitive deps stale in the lock and `npm ci` breaks on Linux CI. +- Report schema version is `1`. Default report dir is `./.twd/run`. Default merged output is `./.twd/merged-run.json`. +- Final version is `1.5.0-beta.0`, published under the `beta` dist-tag. + +## File Structure + +**Create:** + +| File | Responsibility | +|---|---| +| `src/shard.js` | Parse `/`; round-robin id slicing | +| `src/runReport.js` | Build the report object; fingerprint the discovered test list. Pure, no I/O | +| `src/reportFiles.js` | Write a shard's report + coverage; discover and read shard dirs | +| `src/mergeCoverage.js` | Merge Istanbul coverage objects | +| `src/mergeReports.js` | Associative structural merge + consistency validation + derived totals | +| `src/mergeCommand.js` | Orchestrate `merge`: read, merge, write, render, decide exit code | +| `tests/shard.test.js`, `tests/runReport.test.js`, `tests/reportFiles.test.js`, `tests/mergeCoverage.test.js`, `tests/mergeReports.test.js`, `tests/mergeCommand.test.js` | One per module | + +**Modify:** + +| File | Change | +|---|---| +| `src/parseArgs.js` | `--shard`, `--report-dir`; new `parseMergeArgs` | +| `src/index.js` | Slice ids; build and write the report; gate the two behavior changes on `sharded` | +| `src/testSummary.js` | Optional `shards` / `computeMs` params for the merged breakdown | +| `bin/twd-cli.js` | `merge` subcommand + help text | +| `tests/parseArgs.test.js` | 8 full-object assertions gain the new keys | +| `tests/runTests.test.js` | Shard slicing, report writing, and the two non-regression assertions | +| `tests/testSummary.test.js` | Breakdown line rendering | +| `package.json` | `istanbul-lib-coverage` dependency; version `1.5.0-beta.0` | +| `.github/workflows/e2e.yml` | A 2-shard + merge job | +| `README.md`, `CHANGELOG.md` | Document the flags and the command | + +**Why `mergeCommand.js` is separate from `mergeReports.js`:** `mergeReports` must stay pure and associative to be property-testable. Completeness checking (is `1..total` fully covered?) cannot live there, because a partial merge of 2 of 3 shards is a legal intermediate value — enforcing completeness inside the merge would make `merge(merge(a,b),c)` throw. So `mergeReports` validates *consistency* (things preserved under partial merge) and `mergeCommand` enforces *completeness*. + +--- + +### Task 1: Shard spec parsing and id slicing + +**Files:** +- Create: `src/shard.js` +- Test: `tests/shard.test.js` + +**Interfaces:** +- Consumes: nothing. +- Produces: `parseShardSpec(value) -> { index: number, total: number }` (throws `Error` on invalid input); `selectShardIds(ids: string[], index: number, total: number) -> string[]`. + +- [ ] **Step 1: Write the failing test** + +Create `tests/shard.test.js`: + +```js +import { describe, it, expect } from 'vitest'; +import { parseShardSpec, selectShardIds } from '../src/shard.js'; + +describe('parseShardSpec', () => { + it('parses /', () => { + expect(parseShardSpec('2/4')).toEqual({ index: 2, total: 4 }); + expect(parseShardSpec('1/1')).toEqual({ index: 1, total: 1 }); + expect(parseShardSpec(' 3/4 ')).toEqual({ index: 3, total: 4 }); + }); + + // Unlike --record-speed, a bad --shard must never be silently ignored: it + // would run zero tests and exit 0, reading as a green build that tested + // nothing. + it('throws when the index exceeds the total', () => { + expect(() => parseShardSpec('5/4')).toThrow(/between 1 and 4/); + }); + + it('throws on a zero or negative index', () => { + expect(() => parseShardSpec('0/4')).toThrow(/between 1 and 4/); + expect(() => parseShardSpec('-1/4')).toThrow(/Expected \//); + }); + + it('throws on a zero total', () => { + expect(() => parseShardSpec('2/0')).toThrow(/at least 1/); + }); + + it('throws on unparseable input', () => { + for (const bad of ['abc', '', '2', '2/', '/4', '2.5/4', '2/4/6', undefined, null]) { + expect(() => parseShardSpec(bad)).toThrow(/Expected \//); + } + }); + + it('names the offending value in the message', () => { + expect(() => parseShardSpec('9/2')).toThrow(/"9\/2"/); + }); +}); + +describe('selectShardIds', () => { + const ids = Array.from({ length: 12 }, (_, i) => `t${i}`); + + it('takes every Nth id at its own offset', () => { + expect(selectShardIds(ids, 1, 4)).toEqual(['t0', 't4', 't8']); + expect(selectShardIds(ids, 2, 4)).toEqual(['t1', 't5', 't9']); + expect(selectShardIds(ids, 4, 4)).toEqual(['t3', 't7', 't11']); + }); + + it('returns everything when total is 1', () => { + expect(selectShardIds(ids, 1, 1)).toEqual(ids); + }); + + // 3 tests across 4 shards leaves the fourth with nothing. Legal, not an error. + it('returns an empty slice when there are fewer ids than shards', () => { + expect(selectShardIds(['a', 'b', 'c'], 4, 4)).toEqual([]); + expect(selectShardIds([], 1, 4)).toEqual([]); + }); + + // The property that makes sharding trustworthy: nothing lost, nothing doubled. + it('partitions the input — every id lands in exactly one shard', () => { + const many = Array.from({ length: 37 }, (_, i) => `t${i}`); + const total = 5; + const slices = Array.from({ length: total }, (_, i) => selectShardIds(many, i + 1, total)); + const counts = new Map(); + for (const id of slices.flat()) counts.set(id, (counts.get(id) ?? 0) + 1); + expect(counts.size).toBe(many.length); + expect([...counts.values()]).toEqual(many.map(() => 1)); + }); + + it('does not mutate its input', () => { + const input = ['a', 'b', 'c']; + selectShardIds(input, 1, 2); + expect(input).toEqual(['a', 'b', 'c']); + }); +}); +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `npx vitest --run tests/shard.test.js` +Expected: FAIL — `Failed to load ../src/shard.js`. + +- [ ] **Step 3: Write the implementation** + +Create `src/shard.js`: + +```js +// Parses a "/" shard spec, e.g. "2/4" for job 2 of 4. +// +// This throws where src/parseArgs.js silently ignores a malformed +// --record-speed, and the divergence is deliberate: "--shard 5/4" would select +// no tests and exit 0, which reads as a green build that tested nothing. +export function parseShardSpec(value) { + const raw = typeof value === 'string' ? value.trim() : value; + const match = /^(\d+)\/(\d+)$/.exec(String(raw ?? '')); + if (!match) { + throw new Error( + `Invalid --shard "${value}". Expected /, e.g. --shard 2/4.` + ); + } + + const index = Number(match[1]); + const total = Number(match[2]); + + if (total < 1) { + throw new Error(`Invalid --shard "${value}". Total must be at least 1.`); + } + if (index < 1 || index > total) { + throw new Error( + `Invalid --shard "${value}". Index must be between 1 and ${total}.` + ); + } + + return { index, total }; +} + +// Round-robin slice of an ordered id list. `index` is 1-based. +// +// Round-robin rather than contiguous: it balances better when adjacent tests +// have similar cost, and nothing in twd-js requires a suite to run +// contiguously, so locality buys nothing. An empty result is a legal outcome — +// 3 tests across 4 shards leaves the fourth with nothing to run. +export function selectShardIds(ids, index, total) { + return ids.filter((_, i) => i % total === index - 1); +} +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `npx vitest --run tests/shard.test.js` +Expected: PASS, 9 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/shard.js tests/shard.test.js +git commit -m "feat(shard): parse shard specs and slice test ids round-robin" +``` + +--- + +### Task 2: `--shard` and `--report-dir` flags, plus `parseMergeArgs` + +**Files:** +- Modify: `src/parseArgs.js` +- Modify: `tests/parseArgs.test.js` (8 existing assertions at lines 6, 10, 17, 24, 31, 35, 71, 93) +- Test: `tests/parseArgs.test.js` + +**Interfaces:** +- Consumes: `parseShardSpec` from `src/shard.js` (Task 1). +- Produces: `parseRunArgs(argv) -> { testFilters: string[], record: object, shard: {index,total}|null, reportDir: string|null }`; `parseMergeArgs(argv) -> { dir: string|null, out: string|null }`. + +`parseRunArgs` now always returns `shard` and `reportDir`, defaulting to `null`. That is why the 8 existing full-object assertions must be updated — they use `toEqual` on the whole return value. + +- [ ] **Step 1: Update the 8 existing assertions** + +In `tests/parseArgs.test.js`, add `shard: null, reportDir: null` to every full-object `toEqual`. The two single-line ones become: + +```js +// line 6 +expect(parseRunArgs([])).toEqual({ testFilters: [], record: {}, shard: null, reportDir: null }); +// line 31 +expect(parseRunArgs(['--test'])).toEqual({ testFilters: [], record: {}, shard: null, reportDir: null }); +``` + +The six multi-line ones (lines 10, 17, 24, 35, 71, 93) each gain two properties, e.g.: + +```js +expect(parseRunArgs(['--test', 'shows error'])).toEqual({ + testFilters: ['shows error'], + record: {}, + shard: null, + reportDir: null, +}); +``` + +- [ ] **Step 2: Write the failing tests for the new flags** + +Append to `tests/parseArgs.test.js`: + +```js +describe('parseRunArgs shard and report flags', () => { + it('parses --shard in both forms', () => { + expect(parseRunArgs(['--shard', '2/4']).shard).toEqual({ index: 2, total: 4 }); + expect(parseRunArgs(['--shard=3/4']).shard).toEqual({ index: 3, total: 4 }); + }); + + it('throws on an invalid --shard instead of ignoring it', () => { + expect(() => parseRunArgs(['--shard', '5/4'])).toThrow(/Invalid --shard/); + expect(() => parseRunArgs(['--shard=abc'])).toThrow(/Invalid --shard/); + }); + + it('throws on a trailing --shard with no value', () => { + expect(() => parseRunArgs(['--shard'])).toThrow(/Invalid --shard/); + }); + + it('parses --report-dir in both forms', () => { + expect(parseRunArgs(['--report-dir', './out']).reportDir).toBe('./out'); + expect(parseRunArgs(['--report-dir=./out']).reportDir).toBe('./out'); + }); + + it('ignores a trailing --report-dir with no value', () => { + expect(parseRunArgs(['--report-dir']).reportDir).toBeNull(); + }); + + it('combines --shard with --test filters and record flags', () => { + expect(parseRunArgs(['--shard', '2/4', '--test', 'Login', '--record'])).toEqual({ + testFilters: ['Login'], + record: { enabled: true }, + shard: { index: 2, total: 4 }, + reportDir: null, + }); + }); +}); + +describe('parseMergeArgs', () => { + it('reads the directory as the first positional', () => { + expect(parseMergeArgs(['.twd/shards'])).toEqual({ dir: '.twd/shards', out: null }); + }); + + it('parses --out in both forms', () => { + expect(parseMergeArgs(['.twd/shards', '--out', 'merged.json'])) + .toEqual({ dir: '.twd/shards', out: 'merged.json' }); + expect(parseMergeArgs(['.twd/shards', '--out=merged.json'])) + .toEqual({ dir: '.twd/shards', out: 'merged.json' }); + }); + + it('returns a null dir when none is given', () => { + expect(parseMergeArgs([])).toEqual({ dir: null, out: null }); + expect(parseMergeArgs(['--out=merged.json'])).toEqual({ dir: null, out: 'merged.json' }); + }); + + it('takes only the first positional as the directory', () => { + expect(parseMergeArgs(['a', 'b']).dir).toBe('a'); + }); +}); +``` + +Update the import at the top of the file: + +```js +import { parseRunArgs, parseMergeArgs } from "../src/parseArgs.js"; +``` + +- [ ] **Step 3: Run the tests to verify they fail** + +Run: `npx vitest --run tests/parseArgs.test.js` +Expected: FAIL — `parseMergeArgs is not a function`, and the new shard assertions fail on `undefined`. + +- [ ] **Step 4: Implement the flags** + +In `src/parseArgs.js`, add the import at the top: + +```js +import { parseShardSpec } from './shard.js'; +``` + +Inside `parseRunArgs`, add two declarations next to the existing ones: + +```js +export function parseRunArgs(argv) { + const testFilters = []; + const record = {}; + let shard = null; + let reportDir = null; +``` + +Add two branches to the token loop, after the `--test` branch: + +```js + } else if (token === '--shard' || token.startsWith('--shard=')) { + const { value, consumed } = readValue(token, '--shard', i); + // Throws on a malformed spec. A silently-ignored --shard would run zero + // tests and exit 0. + shard = parseShardSpec(value); + i += consumed - 1; + } else if (token === '--report-dir' || token.startsWith('--report-dir=')) { + const { value, consumed } = readValue(token, '--report-dir', i); + if (value !== undefined) reportDir = value; + i += consumed - 1; +``` + +Change the return statement: + +```js + return { testFilters, record, shard, reportDir }; +} +``` + +Append `parseMergeArgs` to the same file: + +```js +// `twd-cli merge [--out ]`. The directory is the first positional +// token; anything after the first is ignored. +export function parseMergeArgs(argv) { + let dir = null; + let out = null; + + const readValue = (token, prefix, index) => { + if (token === prefix) { + return { value: argv[index + 1], consumed: argv[index + 1] !== undefined ? 2 : 1 }; + } + return { value: token.slice(prefix.length + 1), consumed: 1 }; + }; + + for (let i = 0; i < argv.length; i++) { + const token = argv[i]; + + if (token === '--out' || token.startsWith('--out=')) { + const { value, consumed } = readValue(token, '--out', i); + if (value !== undefined) out = value; + i += consumed - 1; + } else if (!token.startsWith('--') && dir === null) { + dir = token; + } + } + + return { dir, out }; +} +``` + +- [ ] **Step 5: Run the tests to verify they pass** + +Run: `npx vitest --run tests/parseArgs.test.js` +Expected: PASS — the 17 original tests plus 10 new ones. + +- [ ] **Step 6: Commit** + +```bash +git add src/parseArgs.js tests/parseArgs.test.js +git commit -m "feat(cli): add --shard, --report-dir, and merge arg parsing" +``` + +--- + +### Task 3: Build the run report and fingerprint discovery + +**Files:** +- Create: `src/runReport.js` +- Test: `tests/runReport.test.js` + +**Interfaces:** +- Consumes: nothing. +- Produces: `REPORT_SCHEMA_VERSION` (number, `1`); `fingerprintTests(orderedIds: string[], filters: string[]) -> string`; `buildRunReport(options) -> report`. `buildRunReport` takes `{ shard, startedAt, endedAt, allTestIds, filters, handlers, tests, executed, notRun, stoppedEarly, coverageFile, recording, contracts }` where `startedAt`/`endedAt` are epoch milliseconds and `shard` is `{index,total}`. + +- [ ] **Step 1: Write the failing test** + +Create `tests/runReport.test.js`: + +```js +import { describe, it, expect } from 'vitest'; +import { buildRunReport, fingerprintTests, REPORT_SCHEMA_VERSION } from '../src/runReport.js'; + +const handlers = [ + { id: 's1', name: 'Login', parent: null, type: 'suite' }, + { id: 't1', name: 'works', parent: 's1', type: 'test' }, +]; + +function build(overrides = {}) { + return buildRunReport({ + shard: { index: 2, total: 4 }, + startedAt: 1_000, + endedAt: 4_500, + allTestIds: ['t1', 't2', 't3'], + filters: [], + handlers, + tests: [{ id: 't1', status: 'pass' }], + executed: 1, + notRun: 0, + stoppedEarly: false, + ...overrides, + }); +} + +describe('fingerprintTests', () => { + it('is stable for the same input', () => { + expect(fingerprintTests(['a', 'b'])).toBe(fingerprintTests(['a', 'b'])); + }); + + it('is prefixed with the algorithm', () => { + expect(fingerprintTests(['a'])).toMatch(/^sha256:[0-9a-f]{64}$/); + }); + + // Order matters: round-robin slicing is only correct if every shard sees the + // same list in the same order. + it('changes when the id order changes', () => { + expect(fingerprintTests(['a', 'b'])).not.toBe(fingerprintTests(['b', 'a'])); + }); + + it('changes when the id set changes', () => { + expect(fingerprintTests(['a', 'b'])).not.toBe(fingerprintTests(['a', 'b', 'c'])); + }); + + it('changes when the filters differ', () => { + expect(fingerprintTests(['a'], ['Login'])).not.toBe(fingerprintTests(['a'], ['Cart'])); + expect(fingerprintTests(['a'], [])).not.toBe(fingerprintTests(['a'], ['Login'])); + }); + + // Filters are OR'd, so their order is not meaningful and must not split + // otherwise-identical shards. + it('ignores the order the filters were given in', () => { + expect(fingerprintTests(['a'], ['Login', 'Cart'])) + .toBe(fingerprintTests(['a'], ['Cart', 'Login'])); + }); +}); + +describe('buildRunReport', () => { + it('stamps the schema version', () => { + expect(build().schemaVersion).toBe(REPORT_SCHEMA_VERSION); + }); + + it('wraps a single shard descriptor in an array', () => { + const report = build(); + expect(report.shards).toHaveLength(1); + expect(report.shards[0]).toMatchObject({ + index: 2, total: 4, executed: 1, notRun: 0, failed: 0, stoppedEarly: false, + }); + }); + + it('derives durationMs and ISO timestamps from epoch millis', () => { + const shard = build().shards[0]; + expect(shard.durationMs).toBe(3500); + expect(shard.startedAt).toBe(new Date(1_000).toISOString()); + expect(shard.endedAt).toBe(new Date(4_500).toISOString()); + }); + + it('counts this shard\'s failures', () => { + const report = build({ + tests: [ + { id: 't1', status: 'pass' }, + { id: 't2', status: 'fail', error: 'boom' }, + { id: 't3', status: 'skip' }, + ], + }); + expect(report.shards[0].failed).toBe(1); + }); + + it('records total discovered tests and the fingerprint', () => { + const report = build(); + expect(report.discovery.totalTests).toBe(3); + expect(report.discovery.fingerprint).toBe(fingerprintTests(['t1', 't2', 't3'], [])); + }); + + it('carries handlers and tests through untouched', () => { + const report = build(); + expect(report.handlers).toEqual(handlers); + expect(report.tests).toEqual([{ id: 't1', status: 'pass' }]); + }); + + it('copies the filters rather than aliasing them', () => { + const filters = ['Login']; + const report = build({ filters }); + filters.push('Cart'); + expect(report.selection.filters).toEqual(['Login']); + }); + + it('defaults contracts to an unconfigured empty block', () => { + expect(build().contracts).toEqual({ + configured: false, partial: false, results: [], skipped: [], + }); + }); + + it('passes a contracts block through when given', () => { + const contracts = { configured: true, partial: true, results: [{ alias: 'a' }], skipped: [] }; + expect(build({ contracts }).contracts).toEqual(contracts); + }); + + it('defaults coverageFile and recording to null', () => { + const shard = build().shards[0]; + expect(shard.coverageFile).toBeNull(); + expect(shard.recording).toBeNull(); + }); +}); +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `npx vitest --run tests/runReport.test.js` +Expected: FAIL — `Failed to load ../src/runReport.js`. + +- [ ] **Step 3: Write the implementation** + +Create `src/runReport.js`: + +```js +import crypto from 'node:crypto'; + +export const REPORT_SCHEMA_VERSION = 1; + +/** + * Hash of the full ordered test id list plus any active --test filters. + * + * Round-robin sharding is correct only if every job enumerates an identical + * test set in an identical order. That breaks silently if the app registers + * tests conditionally — a feature flag, a date, Math.random — or if two shard + * jobs did not build the same code: tests quietly never run and the build stays + * green. Shards compare fingerprints at merge time so it becomes an error. + * + * Filters are OR'd, so their order carries no meaning and is normalized away. + */ +export function fingerprintTests(orderedIds, filters = []) { + const payload = JSON.stringify({ + orderedIds, + filters: [...filters].sort(), + }); + const digest = crypto.createHash('sha256').update(payload).digest('hex'); + return `sha256:${digest}`; +} + +/** + * Assembles the on-disk run report. Pure: no I/O, no clock reads. + * + * `shards` is an array even for a single run, because a merged report has the + * same shape as a single-shard one. That is what makes merging associative and + * lets one set of formatters render both. + * + * `startedAt` and `endedAt` are epoch milliseconds; the report stores ISO + * strings plus the derived duration. + */ +export function buildRunReport({ + shard, + startedAt, + endedAt, + allTestIds, + filters = [], + handlers, + tests, + executed, + notRun, + stoppedEarly, + coverageFile = null, + recording = null, + contracts = null, +}) { + return { + schemaVersion: REPORT_SCHEMA_VERSION, + shards: [ + { + index: shard.index, + total: shard.total, + startedAt: new Date(startedAt).toISOString(), + endedAt: new Date(endedAt).toISOString(), + durationMs: endedAt - startedAt, + executed, + notRun, + // Merged reports do not record which shard ran a test, so the per-shard + // breakdown line could not be rendered without this count. + failed: tests.filter((t) => t.status === 'fail').length, + stoppedEarly, + coverageFile, + recording, + }, + ], + discovery: { + totalTests: allTestIds.length, + fingerprint: fingerprintTests(allTestIds, filters), + }, + selection: { filters: [...filters] }, + handlers, + tests, + contracts: contracts ?? { + configured: false, + partial: false, + results: [], + skipped: [], + }, + }; +} +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `npx vitest --run tests/runReport.test.js` +Expected: PASS, 16 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/runReport.js tests/runReport.test.js +git commit -m "feat(report): build run reports and fingerprint discovered tests" +``` + +--- + +### Task 4: Read and write shard report files + +**Files:** +- Create: `src/reportFiles.js` +- Test: `tests/reportFiles.test.js` + +**Interfaces:** +- Consumes: nothing. +- Produces: `DEFAULT_REPORT_DIR` (`'./.twd/run'`), `DEFAULT_MERGED_OUT` (`'./.twd/merged-run.json'`), `RUN_REPORT_FILE` (`'run.json'`), `COVERAGE_FILE` (`'coverage.json'`); `writeRunReport(dir, report, coverage) -> { reportPath, coveragePath|null }`; `readShardReports(dir) -> Array<{ dir, report }>`; `readShardCoverage(shardDir, coverageFile) -> object|null`. + +- [ ] **Step 1: Write the failing test** + +Create `tests/reportFiles.test.js`: + +```js +import { describe, it, expect, vi, beforeEach } from 'vitest'; + +vi.mock('fs'); + +import fs from 'fs'; +import { + writeRunReport, + readShardReports, + readShardCoverage, + RUN_REPORT_FILE, + COVERAGE_FILE, + DEFAULT_REPORT_DIR, +} from '../src/reportFiles.js'; + +const report = { schemaVersion: 1, tests: [] }; + +describe('writeRunReport', () => { + beforeEach(() => { + vi.clearAllMocks(); + }); + + it('creates the directory recursively', () => { + writeRunReport('.twd/run', report, null); + expect(fs.mkdirSync).toHaveBeenCalledWith('.twd/run', { recursive: true }); + }); + + it('writes pretty-printed JSON so the report is readable by eye', () => { + writeRunReport('.twd/run', report, null); + const [file, body] = vi.mocked(fs.writeFileSync).mock.calls[0]; + expect(file).toBe(`.twd/run/${RUN_REPORT_FILE}`); + expect(body).toBe(`${JSON.stringify(report, null, 2)}\n`); + }); + + it('does not write a coverage file when there is no coverage', () => { + writeRunReport('.twd/run', report, null); + expect(fs.writeFileSync).toHaveBeenCalledTimes(1); + expect(writeRunReport('.twd/run', report, null).coveragePath).toBeNull(); + }); + + // Coverage stays raw and unformatted: it is machine input for nyc, routinely + // several megabytes, and pretty-printing it would double the artifact size. + it('writes coverage compactly alongside the report', () => { + const coverage = { '/a.js': { s: { 0: 1 } } }; + const result = writeRunReport('.twd/run', report, coverage); + expect(fs.writeFileSync).toHaveBeenCalledWith( + `.twd/run/${COVERAGE_FILE}`, + JSON.stringify(coverage), + ); + expect(result.coveragePath).toBe(`.twd/run/${COVERAGE_FILE}`); + }); +}); + +describe('readShardReports', () => { + beforeEach(() => { + vi.clearAllMocks(); + }); + + // This is download-artifact's layout: one directory per artifact name. + it('reads run.json from each child directory', () => { + vi.mocked(fs.existsSync).mockImplementation((p) => String(p).endsWith(RUN_REPORT_FILE)); + vi.mocked(fs.readdirSync).mockReturnValue([ + { name: 'twd-run-1', isDirectory: () => true }, + { name: 'twd-run-2', isDirectory: () => true }, + { name: 'notes.txt', isDirectory: () => false }, + ]); + vi.mocked(fs.readFileSync).mockReturnValue(JSON.stringify(report)); + + const found = readShardReports('.twd/shards'); + + expect(found.map((f) => f.dir)).toEqual([ + '.twd/shards', + '.twd/shards/twd-run-1', + '.twd/shards/twd-run-2', + ]); + expect(found[0].report).toEqual(report); + }); + + it('skips child directories with no run.json', () => { + vi.mocked(fs.existsSync).mockImplementation((p) => String(p).includes('twd-run-1')); + vi.mocked(fs.readdirSync).mockReturnValue([ + { name: 'twd-run-1', isDirectory: () => true }, + { name: 'empty', isDirectory: () => true }, + ]); + vi.mocked(fs.readFileSync).mockReturnValue(JSON.stringify(report)); + + expect(readShardReports('.twd/shards').map((f) => f.dir)) + .toEqual(['.twd/shards/twd-run-1']); + }); + + it('returns an empty array when the directory does not exist', () => { + vi.mocked(fs.existsSync).mockReturnValue(false); + vi.mocked(fs.readdirSync).mockImplementation(() => { + throw Object.assign(new Error('ENOENT'), { code: 'ENOENT' }); + }); + expect(readShardReports('.twd/nope')).toEqual([]); + }); + + it('explains which file failed to parse', () => { + vi.mocked(fs.existsSync).mockReturnValue(true); + vi.mocked(fs.readdirSync).mockReturnValue([]); + vi.mocked(fs.readFileSync).mockReturnValue('{ not json'); + expect(() => readShardReports('.twd/shards')).toThrow(/run\.json/); + }); +}); + +describe('readShardCoverage', () => { + beforeEach(() => { + vi.clearAllMocks(); + }); + + it('reads the named coverage file from the shard directory', () => { + vi.mocked(fs.existsSync).mockReturnValue(true); + vi.mocked(fs.readFileSync).mockReturnValue('{"/a.js":{}}'); + expect(readShardCoverage('.twd/shards/twd-run-1', COVERAGE_FILE)).toEqual({ '/a.js': {} }); + }); + + it('returns null when the shard recorded no coverage file', () => { + expect(readShardCoverage('.twd/shards/twd-run-1', null)).toBeNull(); + expect(fs.existsSync).not.toHaveBeenCalled(); + }); + + // A shard whose coverage file is absent simply does not contribute; merge + // reports the contributor count rather than failing. + it('returns null when the file is missing on disk', () => { + vi.mocked(fs.existsSync).mockReturnValue(false); + expect(readShardCoverage('.twd/shards/twd-run-1', COVERAGE_FILE)).toBeNull(); + }); +}); + +describe('defaults', () => { + it('defaults the report dir to .twd/run', () => { + expect(DEFAULT_REPORT_DIR).toBe('./.twd/run'); + }); +}); +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `npx vitest --run tests/reportFiles.test.js` +Expected: FAIL — `Failed to load ../src/reportFiles.js`. + +- [ ] **Step 3: Write the implementation** + +Create `src/reportFiles.js`: + +```js +import fs from 'fs'; +import path from 'path'; + +export const DEFAULT_REPORT_DIR = './.twd/run'; +export const DEFAULT_MERGED_OUT = './.twd/merged-run.json'; +export const RUN_REPORT_FILE = 'run.json'; +export const COVERAGE_FILE = 'coverage.json'; + +/** + * Writes one shard's report, and its coverage when it collected any. + * + * The report is pretty-printed because a human reads it when a merge complains. + * Coverage is not: it is machine input for nyc, routinely several megabytes, and + * indenting it would roughly double the artifact size for no benefit. + */ +export function writeRunReport(dir, report, coverage = null) { + fs.mkdirSync(dir, { recursive: true }); + + const reportPath = path.join(dir, RUN_REPORT_FILE); + fs.writeFileSync(reportPath, `${JSON.stringify(report, null, 2)}\n`); + + let coveragePath = null; + if (coverage) { + coveragePath = path.join(dir, COVERAGE_FILE); + fs.writeFileSync(coveragePath, JSON.stringify(coverage)); + } + + return { reportPath, coveragePath }; +} + +function readJson(file, label) { + const raw = fs.readFileSync(file, 'utf-8'); + try { + return JSON.parse(raw); + } catch (err) { + throw new Error(`Could not parse ${label} at ${file}: ${err.message}`); + } +} + +/** + * Finds every shard report under `dir`. + * + * actions/download-artifact lays each artifact out as its own directory, so the + * normal shape is `//run.json`. A bare `/run.json` is + * also accepted, which is what a local single-shard run produces. + */ +export function readShardReports(dir) { + const found = []; + + const direct = path.join(dir, RUN_REPORT_FILE); + if (fs.existsSync(direct)) { + found.push({ dir, report: readJson(direct, RUN_REPORT_FILE) }); + } + + let entries; + try { + entries = fs.readdirSync(dir, { withFileTypes: true }); + } catch { + // Missing or unreadable directory: the caller reports "no reports found", + // which is a better message than an ENOENT stack. + return found; + } + + for (const entry of entries) { + if (!entry.isDirectory()) continue; + const shardDir = path.join(dir, entry.name); + const file = path.join(shardDir, RUN_REPORT_FILE); + if (fs.existsSync(file)) { + found.push({ dir: shardDir, report: readJson(file, RUN_REPORT_FILE) }); + } + } + + return found; +} + +/** + * Reads a shard's coverage, or null when it collected none. + * + * Absence is a normal outcome, not an error: a filtered run skips coverage + * entirely. Merge reports how many shards contributed. + */ +export function readShardCoverage(shardDir, coverageFile) { + if (!coverageFile) return null; + const file = path.join(shardDir, coverageFile); + if (!fs.existsSync(file)) return null; + return readJson(file, COVERAGE_FILE); +} +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `npx vitest --run tests/reportFiles.test.js` +Expected: PASS, 11 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/reportFiles.js tests/reportFiles.test.js +git commit -m "feat(report): read and write shard report artifacts" +``` + +--- + +### Task 5: Merge Istanbul coverage objects + +**Files:** +- Create: `src/mergeCoverage.js` +- Modify: `package.json` (promote `istanbul-lib-coverage` to a runtime dependency) +- Modify: `package-lock.json` (regenerated) +- Test: `tests/mergeCoverage.test.js` + +**Interfaces:** +- Consumes: nothing. +- Produces: `mergeCoverage(coverageObjects: Array) -> object` — an Istanbul coverage map JSON with counts summed across inputs. Nulls are skipped. + +`istanbul-lib-coverage@3.2.2` is currently present only transitively via the `@vitest/coverage-v8` devDependency. It must become a real dependency, because `merge` runs in a consumer's project where devDependencies are not installed. + +**This test file must not mock `fs`** — it exercises a real library against in-memory objects. + +- [ ] **Step 1: Add the dependency** + +```bash +npm install istanbul-lib-coverage@^3.2.2 --save +``` + +Verify it landed under `dependencies` (not `devDependencies`): + +```bash +node -p "require('./package.json').dependencies['istanbul-lib-coverage']" +``` + +Expected: `^3.2.2` + +- [ ] **Step 2: Regenerate the lockfile for Linux** + +Docker must be running. + +```bash +npm run lock:linux +``` + +This is mandatory after any dependency change: npm on macOS never installs the wasm32-wasi optional packages, so it leaves their `@emnapi/*` transitive deps stale in the lock and `npm ci` breaks on Linux CI. + +- [ ] **Step 3: Write the failing test** + +Create `tests/mergeCoverage.test.js`: + +```js +import { describe, it, expect } from 'vitest'; +import { mergeCoverage } from '../src/mergeCoverage.js'; + +// Minimal but structurally valid Istanbul file coverage. istanbul-lib-coverage +// validates the shape, so the maps cannot be omitted. +function fileCoverage(path, statementHits, fnHits = 0) { + return { + path, + statementMap: { 0: { start: { line: 1, column: 0 }, end: { line: 1, column: 10 } } }, + fnMap: { + 0: { + name: 'f', + decl: { start: { line: 1, column: 0 }, end: { line: 1, column: 1 } }, + loc: { start: { line: 1, column: 0 }, end: { line: 3, column: 1 } }, + }, + }, + branchMap: {}, + s: { 0: statementHits }, + f: { 0: fnHits }, + b: {}, + }; +} + +describe('mergeCoverage', () => { + it('sums statement hits for the same file across shards', () => { + const merged = mergeCoverage([ + { '/app/src/a.js': fileCoverage('/app/src/a.js', 1) }, + { '/app/src/a.js': fileCoverage('/app/src/a.js', 2) }, + ]); + expect(merged['/app/src/a.js'].s[0]).toBe(3); + }); + + it('sums function hits for the same file across shards', () => { + const merged = mergeCoverage([ + { '/app/src/a.js': fileCoverage('/app/src/a.js', 1, 4) }, + { '/app/src/a.js': fileCoverage('/app/src/a.js', 1, 5) }, + ]); + expect(merged['/app/src/a.js'].f[0]).toBe(9); + }); + + it('unions files that only one shard touched', () => { + const merged = mergeCoverage([ + { '/app/src/a.js': fileCoverage('/app/src/a.js', 1) }, + { '/app/src/b.js': fileCoverage('/app/src/b.js', 7) }, + ]); + expect(Object.keys(merged).sort()).toEqual(['/app/src/a.js', '/app/src/b.js']); + expect(merged['/app/src/b.js'].s[0]).toBe(7); + }); + + // A shard with no coverage file reads back as null and must not break the merge. + it('skips null and undefined entries', () => { + const merged = mergeCoverage([ + null, + { '/app/src/a.js': fileCoverage('/app/src/a.js', 2) }, + undefined, + ]); + expect(merged['/app/src/a.js'].s[0]).toBe(2); + }); + + it('returns an empty map for no input', () => { + expect(mergeCoverage([])).toEqual({}); + expect(mergeCoverage([null])).toEqual({}); + }); + + it('does not mutate its inputs', () => { + const first = { '/app/src/a.js': fileCoverage('/app/src/a.js', 1) }; + mergeCoverage([first, { '/app/src/a.js': fileCoverage('/app/src/a.js', 5) }]); + expect(first['/app/src/a.js'].s[0]).toBe(1); + }); +}); +``` + +- [ ] **Step 4: Run the test to verify it fails** + +Run: `npx vitest --run tests/mergeCoverage.test.js` +Expected: FAIL — `Failed to load ../src/mergeCoverage.js`. + +- [ ] **Step 5: Write the implementation** + +Create `src/mergeCoverage.js`: + +```js +import libCoverage from 'istanbul-lib-coverage'; + +/** + * Combines per-shard Istanbul coverage into one map. + * + * This is the one artifact that needed no bespoke merge logic: summing hit + * counts across runs of the same code is exactly what CoverageMap.merge does, + * and it is the same operation `nyc merge` performs. + * + * Nulls are skipped rather than rejected — a shard that collected no coverage + * (a filtered run, or one that never loaded instrumented code) reads back as + * null and simply does not contribute. + */ +export function mergeCoverage(coverageObjects) { + const map = libCoverage.createCoverageMap({}); + for (const coverage of coverageObjects) { + if (coverage) map.merge(coverage); + } + return map.toJSON(); +} +``` + +- [ ] **Step 6: Run the test to verify it passes** + +Run: `npx vitest --run tests/mergeCoverage.test.js` +Expected: PASS, 6 tests. + +- [ ] **Step 7: Commit** + +```bash +git add package.json package-lock.json src/mergeCoverage.js tests/mergeCoverage.test.js +git commit -m "feat(coverage): merge per-shard Istanbul coverage maps" +``` + +--- + +### Task 6: Associative report merge with consistency validation + +**Files:** +- Create: `src/mergeReports.js` +- Test: `tests/mergeReports.test.js` + +**Interfaces:** +- Consumes: report objects produced by `buildRunReport` (Task 3). +- Produces: `mergeRunReports(reports: object[]) -> object` (same shape as one report; throws on inconsistency); `findMissingShards(report) -> number[]`; `reportTimings(report) -> { wallMs: number, computeMs: number }`; `reportTotals(report) -> { executed: number, notRun: number, consistent: boolean }`. + +**Critical design point:** completeness (`1..total` all present) is deliberately **not** checked here. A 2-of-3 merge is a legal intermediate value, and rejecting it would make `mergeRunReports([mergeRunReports([a, b]), c])` throw — destroying associativity, which is the property that proves no test is lost or doubled. `findMissingShards` is exported separately and called by the merge *command* (Task 9). + +- [ ] **Step 1: Write the failing test** + +Create `tests/mergeReports.test.js`: + +```js +import { describe, it, expect } from 'vitest'; +import { + mergeRunReports, + findMissingShards, + reportTimings, + reportTotals, +} from '../src/mergeReports.js'; + +const HANDLERS = [ + { id: 's1', name: 'Login', parent: null, type: 'suite' }, + { id: 't1', name: 'a', parent: 's1', type: 'test' }, + { id: 't2', name: 'b', parent: 's1', type: 'test' }, + { id: 't3', name: 'c', parent: 's1', type: 'test' }, +]; + +const FINGERPRINT = 'sha256:deadbeef'; + +function makeReport(index, overrides = {}) { + const { + total = 3, + tests = [{ id: `t${index}`, status: 'pass' }], + startedAt = `2026-08-19T10:00:0${index}.000Z`, + endedAt = `2026-08-19T10:00:1${index}.000Z`, + durationMs = 10_000, + executed = 1, + notRun = 0, + failed = 0, + stoppedEarly = false, + coverageFile = 'coverage.json', + contracts = { configured: true, partial: false, results: [], skipped: [] }, + fingerprint = FINGERPRINT, + schemaVersion = 1, + totalTests = 3, + } = overrides; + + return { + schemaVersion, + shards: [{ + index, total, startedAt, endedAt, durationMs, + executed, notRun, failed, stoppedEarly, coverageFile, recording: null, + }], + discovery: { totalTests, fingerprint }, + selection: { filters: [] }, + handlers: HANDLERS, + tests, + contracts, + }; +} + +describe('mergeRunReports', () => { + it('concatenates tests across shards', () => { + const merged = mergeRunReports([makeReport(1), makeReport(2), makeReport(3)]); + expect(merged.tests.map((t) => t.id)).toEqual(['t1', 't2', 't3']); + }); + + it('sorts shard descriptors by index regardless of input order', () => { + const merged = mergeRunReports([makeReport(3), makeReport(1), makeReport(2)]); + expect(merged.shards.map((s) => s.index)).toEqual([1, 2, 3]); + }); + + it('keeps the single-report shape', () => { + const merged = mergeRunReports([makeReport(1), makeReport(2), makeReport(3)]); + expect(Object.keys(merged).sort()).toEqual( + ['contracts', 'discovery', 'handlers', 'schemaVersion', 'selection', 'shards', 'tests'], + ); + expect(merged.handlers).toEqual(HANDLERS); + expect(merged.discovery).toEqual({ totalTests: 3, fingerprint: FINGERPRINT }); + }); + + // The property that proves nothing is lost or doubled. It only holds because + // completeness is checked outside this function. + it('is associative', () => { + const a = makeReport(1); + const b = makeReport(2); + const c = makeReport(3); + expect(mergeRunReports([mergeRunReports([a, b]), c])) + .toEqual(mergeRunReports([a, b, c])); + expect(mergeRunReports([a, mergeRunReports([b, c])])) + .toEqual(mergeRunReports([a, b, c])); + }); + + it('accepts a partial merge without complaining about gaps', () => { + const merged = mergeRunReports([makeReport(1), makeReport(2)]); + expect(merged.shards.map((s) => s.index)).toEqual([1, 2]); + }); + + it('concatenates contract results and skipped entries', () => { + const merged = mergeRunReports([ + makeReport(1, { contracts: { configured: true, partial: false, results: [{ alias: 'a' }], skipped: [{ alias: 'x' }] } }), + makeReport(2, { contracts: { configured: true, partial: false, results: [{ alias: 'b' }], skipped: [] } }), + ]); + expect(merged.contracts.results).toEqual([{ alias: 'a' }, { alias: 'b' }]); + expect(merged.contracts.skipped).toEqual([{ alias: 'x' }]); + }); + + it('ORs the contracts partial flag', () => { + const partial = makeReport(2, { contracts: { configured: true, partial: true, results: [], skipped: [] } }); + expect(mergeRunReports([makeReport(1), partial]).contracts.partial).toBe(true); + expect(mergeRunReports([makeReport(1), makeReport(2)]).contracts.partial).toBe(false); + }); + + it('throws on an empty input', () => { + expect(() => mergeRunReports([])).toThrow(/No shard reports/); + }); + + it('throws when schema versions disagree', () => { + expect(() => mergeRunReports([makeReport(1), makeReport(2, { schemaVersion: 2 })])) + .toThrow(/schemaVersion/); + }); + + // The safety net: shards that saw different test sets must never be combined. + it('throws when fingerprints disagree', () => { + expect(() => mergeRunReports([makeReport(1), makeReport(2, { fingerprint: 'sha256:other' })])) + .toThrow(/different test sets/); + }); + + it('throws when shard totals disagree', () => { + expect(() => mergeRunReports([makeReport(1), makeReport(2, { total: 4 })])) + .toThrow(/shard total/); + }); + + it('throws when the same shard index appears twice', () => { + expect(() => mergeRunReports([makeReport(1), makeReport(1)])) + .toThrow(/more than once/); + }); + + it('throws when a test id appears in two shards', () => { + expect(() => mergeRunReports([ + makeReport(1, { tests: [{ id: 'dup', status: 'pass' }] }), + makeReport(2, { tests: [{ id: 'dup', status: 'pass' }] }), + ])).toThrow(/"dup" appears in more than one shard/); + }); + + it('does not mutate the input reports', () => { + const a = makeReport(1); + const b = makeReport(2); + mergeRunReports([b, a]); + expect(a.shards).toHaveLength(1); + expect(b.shards[0].index).toBe(2); + }); +}); + +describe('findMissingShards', () => { + it('returns an empty array when every shard is present', () => { + expect(findMissingShards(mergeRunReports([makeReport(1), makeReport(2), makeReport(3)]))) + .toEqual([]); + }); + + it('names the gaps', () => { + expect(findMissingShards(mergeRunReports([makeReport(1), makeReport(3)]))).toEqual([2]); + expect(findMissingShards(mergeRunReports([makeReport(2)]))).toEqual([1, 3]); + }); +}); + +describe('reportTimings', () => { + // Wall clock is what the developer waited; compute is what was paid for. + it('reports wall clock as the span and compute as the sum', () => { + const merged = mergeRunReports([makeReport(1), makeReport(2), makeReport(3)]); + // starts 10:00:01..03, ends 10:00:11..13 -> span 12s; 3 x 10s compute + expect(reportTimings(merged)).toEqual({ wallMs: 12_000, computeMs: 30_000 }); + }); + + it('makes wall and compute equal for a single shard', () => { + const single = mergeRunReports([makeReport(1, { total: 1 })]); + const { wallMs, computeMs } = reportTimings(single); + expect(wallMs).toBe(computeMs); + }); +}); + +describe('reportTotals', () => { + it('sums executed and notRun and confirms they account for discovery', () => { + const merged = mergeRunReports([makeReport(1), makeReport(2), makeReport(3)]); + expect(reportTotals(merged)).toEqual({ executed: 3, notRun: 0, consistent: true }); + }); + + it('flags totals that do not add up to the discovered count', () => { + const merged = mergeRunReports([makeReport(1), makeReport(2)]); + expect(reportTotals(merged).consistent).toBe(false); + }); + + it('counts a bailed shard\'s notRun', () => { + const merged = mergeRunReports([ + makeReport(1), + makeReport(2, { executed: 1, notRun: 1, stoppedEarly: true, failed: 1 }), + makeReport(3), + ]); + expect(reportTotals(merged)).toEqual({ executed: 3, notRun: 1, consistent: false }); + }); +}); +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `npx vitest --run tests/mergeReports.test.js` +Expected: FAIL — `Failed to load ../src/mergeReports.js`. + +- [ ] **Step 3: Write the implementation** + +Create `src/mergeReports.js`: + +```js +/** + * Combines shard reports into one report of the same shape. + * + * Only *consistency* is validated here — the things that stay true under a + * partial merge. Completeness (is 1..total all present?) is checked by + * findMissingShards, called from the merge command, because a 2-of-3 merge is a + * legal intermediate value: rejecting it here would make + * mergeRunReports([mergeRunReports([a, b]), c]) throw and destroy + * associativity, which is the property that proves no test is lost or doubled. + */ +export function mergeRunReports(reports) { + if (!reports.length) { + throw new Error( + 'No shard reports to merge. Expected /*/run.json or /run.json.' + ); + } + + const versions = [...new Set(reports.map((r) => r.schemaVersion))]; + if (versions.length > 1) { + throw new Error( + `Shard reports disagree on schemaVersion (${versions.sort().join(', ')}). ` + + 'Every shard job must run the same twd-cli version.' + ); + } + + const fingerprints = new Set(reports.map((r) => r.discovery.fingerprint)); + if (fingerprints.size > 1) { + throw new Error( + 'Shard reports discovered different test sets, so they cannot be merged. ' + + 'Either tests are registered conditionally (a feature flag, a date, ' + + 'Math.random), or the shard jobs did not build the same code.' + ); + } + + const shards = reports.flatMap((r) => r.shards); + + const totals = [...new Set(shards.map((s) => s.total))]; + if (totals.length > 1) { + throw new Error( + `Shard reports disagree on shard total (${totals.sort((a, b) => a - b).join(', ')}). ` + + 'Every shard job must pass the same --shard total.' + ); + } + + const byIndex = new Set(); + for (const shard of shards) { + if (byIndex.has(shard.index)) { + throw new Error(`Shard ${shard.index}/${shard.total} appears more than once.`); + } + byIndex.add(shard.index); + } + + const tests = []; + const testIds = new Set(); + for (const report of reports) { + for (const test of report.tests) { + if (testIds.has(test.id)) { + throw new Error( + `Test id "${test.id}" appears in more than one shard — the shard slices overlap.` + ); + } + testIds.add(test.id); + tests.push(test); + } + } + + const first = reports[0]; + + return { + schemaVersion: first.schemaVersion, + // Copy before sorting: sort mutates, and the input reports are the caller's. + shards: [...shards].sort((a, b) => a.index - b.index), + discovery: first.discovery, + selection: first.selection, + // Identical across shards, guaranteed by the fingerprint check above. + handlers: first.handlers, + tests, + contracts: { + configured: first.contracts.configured, + partial: reports.some((r) => r.contracts.partial), + results: reports.flatMap((r) => r.contracts.results), + skipped: reports.flatMap((r) => r.contracts.skipped), + }, + }; +} + +/** + * Shard indices in 1..total that no report accounted for. + * + * A gap almost always means a shard job died before uploading its artifact. It + * must be loud: a silent 3-of-4 merge reads as a complete green run. + */ +export function findMissingShards(report) { + const total = report.shards[0]?.total ?? 0; + const present = new Set(report.shards.map((s) => s.index)); + const missing = []; + for (let i = 1; i <= total; i++) { + if (!present.has(i)) missing.push(i); + } + return missing; +} + +/** + * Wall clock (what the developer waited) and compute (what was paid for). + * + * Derived rather than stored, so the two can never drift out of agreement with + * the per-shard timestamps they come from. + */ +export function reportTimings(report) { + const starts = report.shards.map((s) => Date.parse(s.startedAt)); + const ends = report.shards.map((s) => Date.parse(s.endedAt)); + return { + wallMs: Math.max(...ends) - Math.min(...starts), + computeMs: report.shards.reduce((sum, s) => sum + s.durationMs, 0), + }; +} + +/** + * Executed and not-run totals, plus whether they account for every discovered + * test. An inconsistent result points at a shard-math bug, not user error. + */ +export function reportTotals(report) { + const executed = report.shards.reduce((sum, s) => sum + s.executed, 0); + const notRun = report.shards.reduce((sum, s) => sum + s.notRun, 0); + return { + executed, + notRun, + consistent: executed + notRun === report.discovery.totalTests, + }; +} +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `npx vitest --run tests/mergeReports.test.js` +Expected: PASS, 22 tests. + +- [ ] **Step 5: Commit** + +```bash +git add src/mergeReports.js tests/mergeReports.test.js +git commit -m "feat(merge): associative report merge with consistency validation" +``` + +--- + +### Task 7: Per-shard breakdown in the run summary + +**Files:** +- Modify: `src/testSummary.js` +- Test: `tests/testSummary.test.js` + +**Interfaces:** +- Consumes: `shards` array from a merged report (Task 6), `computeMs` from `reportTimings`. +- Produces: `formatRunComplete({ testStatus, handlers, durationMs, notRun, stoppedEarly, maxFailures, shards, computeMs })` — two new optional params, both defaulting to `null`. + +**Constraint:** with `shards` absent or of length 1, output must be byte-identical to today's. + +- [ ] **Step 1: Write the failing test** + +Append to `tests/testSummary.test.js`: + +```js +describe('formatRunComplete with shards', () => { + const handlers = [ + { id: 's1', name: 'Login', parent: null, type: 'suite' }, + { id: 't1', name: 'works', parent: 's1', type: 'test' }, + ]; + const testStatus = [{ id: 't1', status: 'pass' }]; + + function shard(index, overrides = {}) { + return { index, total: 4, executed: 30, failed: 0, notRun: 0, stoppedEarly: false, ...overrides }; + } + + it('adds a shard breakdown line when more than one shard merged', () => { + const output = formatRunComplete({ + testStatus, handlers, durationMs: 38_200, computeMs: 134_200, + shards: [shard(1), shard(2, { failed: 3 }), shard(3), shard(4)], + }); + expect(output).toContain('Shards: 1 ✓30 | 2 ✗30 | 3 ✓30 | 4 ✓30'); + }); + + it('reports wall clock and compute separately for a merged run', () => { + const output = formatRunComplete({ + testStatus, handlers, durationMs: 38_200, computeMs: 134_200, + shards: [shard(1), shard(2)], + }); + expect(output).toContain('Duration: 38.2s wall | 134.2s across 2 shards'); + }); + + // The existing single-run format must not shift. + it('keeps the plain duration line when there are no shards', () => { + const output = formatRunComplete({ testStatus, handlers, durationMs: 4_200 }); + expect(output).toContain('Duration: 4.2s'); + expect(output).not.toContain('wall'); + expect(output).not.toContain('Shards:'); + }); + + it('keeps the plain duration line for a single shard', () => { + const output = formatRunComplete({ + testStatus, handlers, durationMs: 4_200, computeMs: 4_200, shards: [shard(1, { total: 1 })], + }); + expect(output).toContain('Duration: 4.2s'); + expect(output).not.toContain('Shards:'); + }); +}); +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `npx vitest --run tests/testSummary.test.js` +Expected: FAIL — the breakdown and wall/compute assertions find no such line. + +- [ ] **Step 3: Implement the breakdown** + +In `src/testSummary.js`, extend the destructured parameters: + +```js +export function formatRunComplete({ + testStatus, + handlers, + durationMs, + notRun = 0, + stoppedEarly = false, + maxFailures, + shards = null, + computeMs = null, +}) { +``` + +Replace the existing duration block: + +```js + if (notRun > 0) lines.push(` Not run: ${notRun}`); + lines.push(` Duration: ${duration}s`); +``` + +with: + +```js + if (notRun > 0) lines.push(` Not run: ${notRun}`); + + // A merged run has two meaningful durations: the span the developer waited, + // and the compute it consumed. A single run has only one, and its line must + // stay byte-identical to what it has always printed. + const merged = Array.isArray(shards) && shards.length > 1; + if (merged) { + const compute = (computeMs / 1000).toFixed(1); + lines.push(` Duration: ${duration}s wall | ${compute}s across ${shards.length} shards`); + const cells = shards.map((s) => `${s.index} ${s.failed > 0 ? '✗' : '✓'}${s.executed}`); + lines.push(` Shards: ${cells.join(' | ')}`); + } else { + lines.push(` Duration: ${duration}s`); + } +``` + +- [ ] **Step 4: Run the full suite to verify nothing shifted** + +Run: `npx vitest --run tests/testSummary.test.js` +Expected: PASS — the new tests plus every existing one, unchanged. + +- [ ] **Step 5: Commit** + +```bash +git add src/testSummary.js tests/testSummary.test.js +git commit -m "feat(summary): add per-shard breakdown and wall vs compute duration" +``` + +--- + +### Task 8: Wire sharding and report writing into the run + +**Files:** +- Modify: `src/index.js` +- Test: `tests/runTests.test.js` + +**Interfaces:** +- Consumes: `selectShardIds` (Task 1), `buildRunReport` (Task 3), `writeRunReport` / `DEFAULT_REPORT_DIR` / `COVERAGE_FILE` (Task 4). +- Produces: `runTests({ testFilters, recordOverrides, shard, reportDir }) -> Promise`. Two new options; the return value is unchanged. + +This is the task where the Global Constraint bites hardest. With `shard` absent, every conditional below must reduce to the expression it replaced. + +- [ ] **Step 1: Write the failing tests** + +Append to `tests/runTests.test.js`: + +```js +describe('runTests sharding', () => { + function fourTests() { + return { + handlers: [ + { id: '1', name: 'a', type: 'test' }, + { id: '2', name: 'b', type: 'test' }, + { id: '3', name: 'c', type: 'test' }, + { id: '4', name: 'd', type: 'test' }, + ], + testStatus: [{ id: '2', status: 'pass' }, { id: '4', status: 'pass' }], + }; + } + + function runJson() { + const call = vi.mocked(fs.writeFileSync).mock.calls + .find(([file]) => String(file).endsWith('run.json')); + return call ? JSON.parse(call[1]) : null; + } + + // Round-robin: shard 2 of 2 takes the odd indices, i.e. the 2nd and 4th ids. + it('runs only its round-robin slice', async () => { + const { handlers, testStatus } = fourTests(); + const page = createMockPage({ handlers, testStatus }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + + await runTests({ shard: { index: 2, total: 2 } }); + + expect(page.evaluate).toHaveBeenCalledWith(expect.any(Function), 2, ['2', '4']); + }); + + it('writes a run report to the default report dir', async () => { + const { handlers, testStatus } = fourTests(); + const page = createMockPage({ handlers, testStatus }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + + await runTests({ shard: { index: 2, total: 2 } }); + + expect(fs.mkdirSync).toHaveBeenCalledWith('./.twd/run', { recursive: true }); + const report = runJson(); + expect(report.schemaVersion).toBe(1); + expect(report.shards[0]).toMatchObject({ index: 2, total: 2, executed: 2 }); + expect(report.discovery.totalTests).toBe(4); + expect(report.tests.map((t) => t.id)).toEqual(['2', '4']); + }); + + it('honors --report-dir', async () => { + const { handlers, testStatus } = fourTests(); + const page = createMockPage({ handlers, testStatus }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + + await runTests({ shard: { index: 1, total: 1 }, reportDir: './out' }); + + expect(fs.mkdirSync).toHaveBeenCalledWith('./out', { recursive: true }); + }); + + it('writes no report when not sharded', async () => { + const { handlers, testStatus } = fourTests(); + const page = createMockPage({ handlers, testStatus }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + + await runTests(); + + expect(runJson()).toBeNull(); + }); + + // 3 tests across 4 shards leaves the fourth with nothing to run. It must + // still write a valid report, or merge sees a gap it cannot explain. + it('writes a valid empty report when its slice is empty', async () => { + const handlers = [ + { id: '1', name: 'a', type: 'test' }, + { id: '2', name: 'b', type: 'test' }, + { id: '3', name: 'c', type: 'test' }, + ]; + const page = createMockPage({ handlers, testStatus: [] }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + + const hasFailures = await runTests({ shard: { index: 4, total: 4 } }); + + expect(hasFailures).toBe(false); + const report = runJson(); + expect(report.tests).toEqual([]); + expect(report.shards[0]).toMatchObject({ index: 4, total: 4, executed: 0, failed: 0 }); + // The fingerprint still covers the whole suite, so an empty shard merges + // cleanly with the three that ran something. + expect(report.discovery.totalTests).toBe(3); + }); + + // Filters resolve first, then the filtered list is sharded. A filtered run's + // coverage is a misleading project-wide number, sharded or not. + it('skips coverage when a filter is combined with a shard', async () => { + const { handlers, testStatus } = fourTests(); + const page = createMockPage({ handlers, testStatus }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + vi.mocked(loadConfig).mockReturnValue({ ...defaultMockConfig, coverage: true }); + + await runTests({ shard: { index: 1, total: 2 }, testFilters: ['a'] }); + + const files = vi.mocked(fs.writeFileSync).mock.calls.map(([f]) => String(f)); + expect(files.some((f) => f.endsWith('coverage.json'))).toBe(false); + expect(runJson().shards[0].coverageFile).toBeNull(); + expect(runJson().selection.filters).toEqual(['a']); + }); +}); + +describe('runTests non-regression: non-sharded behavior is unchanged', () => { + // The !hasFailures coverage gate is relaxed only for sharded runs. A plain + // failing run must still write nothing, exactly as in 1.4.0. + it('writes no coverage when a non-sharded run fails', async () => { + const handlers = [{ id: '1', name: 'a', type: 'test' }]; + const page = createMockPage({ handlers, testStatus: [{ id: '1', status: 'fail', error: 'boom' }] }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + vi.mocked(loadConfig).mockReturnValue({ ...defaultMockConfig, coverage: true }); + + await runTests(); + + expect(fs.writeFileSync).not.toHaveBeenCalled(); + }); + + // Likewise the early-stop contract skip. + it('still skips contract validation when a non-sharded run stops early', async () => { + const handlers = [{ id: '1', name: 'a', type: 'test' }]; + const page = createMockPage({ handlers, testStatus: [{ id: '1', status: 'fail', error: 'boom' }] }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + vi.mocked(loadConfig).mockReturnValue({ + ...defaultMockConfig, maxFailures: 1, contracts: [{ source: 'api.json' }], + }); + vi.mocked(loadContracts).mockResolvedValue([]); + + await runTests(); + + expect(validateMocks).not.toHaveBeenCalled(); + }); +}); + +describe('runTests sharded behavior changes', () => { + it('writes coverage for a sharded run that fails', async () => { + const handlers = [{ id: '1', name: 'a', type: 'test' }]; + const page = createMockPage({ handlers, testStatus: [{ id: '1', status: 'fail', error: 'boom' }] }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + vi.mocked(loadConfig).mockReturnValue({ ...defaultMockConfig, coverage: true }); + + await runTests({ shard: { index: 1, total: 1 } }); + + const files = vi.mocked(fs.writeFileSync).mock.calls.map(([f]) => String(f)); + expect(files.some((f) => f.endsWith('coverage.json'))).toBe(true); + // Never the path nyc reads by default: a shard's partial coverage there + // would masquerade as the whole run's. + expect(files.some((f) => f.includes('.nyc_output'))).toBe(false); + }); + + it('validates contracts on a sharded early stop and marks them partial', async () => { + const handlers = [{ id: '1', name: 'a', type: 'test' }]; + const page = createMockPage({ handlers, testStatus: [{ id: '1', status: 'fail', error: 'boom' }] }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + vi.mocked(loadConfig).mockReturnValue({ + ...defaultMockConfig, maxFailures: 1, contracts: [{ source: 'api.json' }], + }); + vi.mocked(loadContracts).mockResolvedValue([]); + vi.mocked(validateMocks).mockReturnValue({ results: [{ alias: 'a' }], skipped: [] }); + vi.mocked(printContractReport).mockReturnValue(false); + + await runTests({ shard: { index: 1, total: 1 } }); + + expect(validateMocks).toHaveBeenCalled(); + const call = vi.mocked(fs.writeFileSync).mock.calls + .find(([file]) => String(file).endsWith('run.json')); + const report = JSON.parse(call[1]); + expect(report.contracts).toMatchObject({ configured: true, partial: true }); + expect(report.contracts.results).toEqual([{ alias: 'a' }]); + }); +}); +``` + +- [ ] **Step 2: Run the tests to verify they fail** + +Run: `npx vitest --run tests/runTests.test.js` +Expected: FAIL — the shard slice assertion sees all four ids, and no `run.json` is written. + +- [ ] **Step 3: Add imports and options** + +In `src/index.js`, add to the import block: + +```js +import { selectShardIds } from './shard.js'; +import { buildRunReport } from './runReport.js'; +import { writeRunReport, DEFAULT_REPORT_DIR, COVERAGE_FILE } from './reportFiles.js'; +``` + +Change the destructure at line 46: + +```js + const { testFilters = [], recordOverrides = {}, shard = null, reportDir = null } = options; + const sharded = Boolean(shard); +``` + +Add one declaration alongside the other `let`s near the top of the function, so the recording details can reach the report: + +```js + let recordingInfo = null; +``` + +- [ ] **Step 4: Slice the ids** + +Replace line 164: + +```js + const baseIds = selectedIds ?? orderedTestIds(registeredHandlers); +``` + +with: + +```js + // The full ordered list, before filtering or slicing. This is what the + // fingerprint hashes and what discovery.totalTests reports, so every shard + // agrees on it regardless of which slice it took. + const allTestIds = orderedTestIds(registeredHandlers); + const filteredIds = selectedIds ?? allTestIds; + const baseIds = sharded + ? selectShardIds(filteredIds, shard.index, shard.total) + : filteredIds; + + if (sharded) { + console.log( + `Shard ${shard.index}/${shard.total}: running ${baseIds.length} of ${filteredIds.length} test(s).` + ); + } +``` + +- [ ] **Step 5: Capture recording details for the report** + +In the recording success branch (around line 277), replace: + +```js + } else { + console.log(`Recorded ${executed} test(s) to ${recordOutput}`); + } +``` + +with: + +```js + } else { + recordingInfo = { file: recordOutput, bytes: recordedFileSize(recordOutputPath) }; + console.log(`Recorded ${executed} test(s) to ${recordOutput}`); + } +``` + +- [ ] **Step 6: Capture the end timestamp** + +Replace line 282: + +```js + const durationMs = Date.now() - startedAt; +``` + +with: + +```js + const endedAt = Date.now(); + const durationMs = endedAt - startedAt; +``` + +- [ ] **Step 7: Gate the contract change on sharding** + +Replace the whole contract block (lines 296-319) with: + +```js + // Contract validation. A sharded run validates even after an early stop and + // flags the result partial, so merge can say exactly what is missing rather + // than silently dropping a shard's worth of mocks. A non-sharded run keeps + // skipping, exactly as before. + const contractsConfigured = Boolean(config.contracts && config.contracts.length > 0); + let contractsBlock = { + configured: contractsConfigured, + partial: false, + results: [], + skipped: [], + }; + + if (contractsConfigured && (sharded || !stoppedEarly)) { + if (collectedMocks.size === 0) { + console.log('\nNo mocks collected — ensure twd-js supports contract collection'); + } + const validationOutput = validateMocks(collectedMocks, contractValidators); + const hasContractErrors = printContractReport(validationOutput); + if (hasContractErrors) { + hasFailures = true; + } + + contractsBlock = { + configured: true, + partial: stoppedEarly, + results: validationOutput.results, + skipped: validationOutput.skipped, + }; + + if (stoppedEarly) { + console.log('\n⚠ Contract data is partial — this shard stopped early.'); + } + + // Only a whole run produces a meaningful markdown report. Under sharding + // each shard would overwrite the others with a quarter of the picture, so + // `merge` writes it instead. + if (config.contractReportPath && !sharded) { + const reportPath = path.resolve(workingDir, config.contractReportPath); + const reportDirPath = path.dirname(reportPath); + if (!fs.existsSync(reportDirPath)) { + fs.mkdirSync(reportDirPath, { recursive: true }); + } + const markdown = generateContractMarkdown(validationOutput); + fs.writeFileSync(reportPath, markdown); + console.log(`Contract report written to ${config.contractReportPath}`); + } + } else if (contractsConfigured && stoppedEarly) { + console.log('\nSkipping contract validation — run stopped early (partial data).'); + } +``` + +- [ ] **Step 8: Gate the coverage change on sharding** + +Replace the whole coverage block (lines 321-344) with: + +```js + // Handle code coverage. + // + // The filter gate is unchanged: a --test filter still suppresses coverage, + // because a filtered run's number is a misleading project-wide figure. A + // shard slice is not a filter. + // + // The failure gate is relaxed for sharded runs only. hasFailures is per + // shard, so applying it here would let three green shards write coverage + // while a red fourth writes none — a merged report that looks complete but + // is missing a quarter of the code paths. `merge` applies the gate to the + // true global result instead. + if (selectedIds && config.coverage) { + console.log('Skipping coverage collection (test filter active).'); + } + + let coverageData = null; + if (config.coverage && !selectedIds && (sharded || !hasFailures)) { + coverageData = await page.evaluate(() => window.__coverage__); + if (!coverageData) { + console.log('No code coverage data found.'); + } + } + + // A sharded run's coverage goes to the report dir and nowhere else. Writing + // it to .nyc_output/out.json — the path nyc reads by default — would let one + // shard's partial data masquerade as the whole run's. + if (coverageData && !sharded) { + const coverageDir = path.resolve(workingDir, config.coverageDir); + const nycDir = path.resolve(workingDir, config.nycOutputDir); + + if (!fs.existsSync(nycDir)) { + fs.mkdirSync(nycDir, { recursive: true }); + } + if (!fs.existsSync(coverageDir)) { + fs.mkdirSync(coverageDir, { recursive: true }); + } + + const coveragePath = path.join(nycDir, 'out.json'); + fs.writeFileSync(coveragePath, JSON.stringify(coverageData)); + console.log(`Code coverage data written to ${coveragePath}`); + } +``` + +- [ ] **Step 9: Write the shard report** + +After the `formatRunComplete` block (line 357) and before `return hasFailures;`, insert: + +```js + // Written last, and only for a sharded run. A run that threw never gets + // here on purpose: its artifact stays absent, and `merge` reports the gap as + // "a shard job likely failed before uploading", which is the accurate + // diagnosis. A half-written report would be a worse lie. + if (sharded) { + const dir = reportDir ?? DEFAULT_REPORT_DIR; + const report = buildRunReport({ + shard, + startedAt, + endedAt, + allTestIds, + filters: testFilters, + handlers, + tests: testStatus, + executed, + notRun, + stoppedEarly, + coverageFile: coverageData ? COVERAGE_FILE : null, + recording: recordingInfo, + contracts: contractsBlock, + }); + const { reportPath } = writeRunReport(dir, report, coverageData); + console.log(`Shard report written to ${reportPath}`); + } +``` + +- [ ] **Step 10: Run the full suite** + +Run: `npm run test:ci` +Expected: PASS — every existing test plus the new ones. The two non-regression tests are the ones that matter most here. + +- [ ] **Step 11: Commit** + +```bash +git add src/index.js tests/runTests.test.js +git commit -m "feat(run): slice tests by shard and write a run report artifact" +``` + +--- + +### Task 9: The `merge` command + +**Files:** +- Create: `src/mergeCommand.js` +- Modify: `bin/twd-cli.js` +- Test: `tests/mergeCommand.test.js` + +**Interfaces:** +- Consumes: `readShardReports` / `readShardCoverage` / `DEFAULT_MERGED_OUT` (Task 4), `mergeCoverage` (Task 5), `mergeRunReports` / `findMissingShards` / `reportTimings` / `reportTotals` (Task 6), `formatRunComplete` (Task 7), plus the existing `loadConfig`, `printContractReport`, `generateContractMarkdown`. +- Produces: `runMerge({ dir, out }) -> boolean` (true when the merged run has failures). Throws on unusable input. + +- [ ] **Step 1: Write the failing test** + +Create `tests/mergeCommand.test.js`: + +```js +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; + +vi.mock('fs'); +vi.mock('../src/config.js', () => ({ loadConfig: vi.fn() })); +vi.mock('../src/contractReport.js', () => ({ printContractReport: vi.fn() })); +vi.mock('../src/reportFiles.js', async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, readShardReports: vi.fn(), readShardCoverage: vi.fn() }; +}); + +import fs from 'fs'; +import { loadConfig } from '../src/config.js'; +import { printContractReport } from '../src/contractReport.js'; +import { readShardReports, readShardCoverage } from '../src/reportFiles.js'; +import { runMerge } from '../src/mergeCommand.js'; + +const HANDLERS = [ + { id: 's1', name: 'Login', parent: null, type: 'suite' }, + { id: 't1', name: 'a', parent: 's1', type: 'test' }, + { id: 't2', name: 'b', parent: 's1', type: 'test' }, +]; + +function shardReport(index, overrides = {}) { + const { total = 2, tests = [{ id: `t${index}`, status: 'pass' }], failed = 0 } = overrides; + return { + schemaVersion: 1, + shards: [{ + index, total, + startedAt: `2026-08-19T10:00:0${index}.000Z`, + endedAt: `2026-08-19T10:00:1${index}.000Z`, + durationMs: 10_000, + executed: 1, notRun: 0, failed, + stoppedEarly: false, coverageFile: 'coverage.json', recording: null, + }], + discovery: { totalTests: 2, fingerprint: 'sha256:same' }, + selection: { filters: [] }, + handlers: HANDLERS, + tests, + contracts: { configured: false, partial: false, results: [], skipped: [] }, + }; +} + +const baseConfig = { + coverage: true, + nycOutputDir: './.nyc_output', + maxFailures: 10, +}; + +function writtenFiles() { + return vi.mocked(fs.writeFileSync).mock.calls.map(([f]) => String(f)); +} + +describe('runMerge', () => { + beforeEach(() => { + vi.clearAllMocks(); + vi.mocked(loadConfig).mockReturnValue({ ...baseConfig }); + vi.mocked(readShardCoverage).mockReturnValue(null); + vi.spyOn(console, 'log').mockImplementation(() => {}); + vi.spyOn(console, 'warn').mockImplementation(() => {}); + }); + + afterEach(() => { + vi.restoreAllMocks(); + }); + + it('requires a directory', () => { + expect(() => runMerge({})).toThrow(/Usage: twd-cli merge/); + }); + + it('errors when no shard reports were found', () => { + vi.mocked(readShardReports).mockReturnValue([]); + expect(() => runMerge({ dir: '.twd/shards' })).toThrow(/No shard reports found/); + }); + + // The failure that must never be silent: three green shards and one that + // never uploaded would otherwise read as a complete green run. + it('errors and names the gap when a shard is missing', () => { + vi.mocked(readShardReports).mockReturnValue([{ dir: 'a', report: shardReport(1) }]); + expect(() => runMerge({ dir: '.twd/shards' })).toThrow(/Missing shard report\(s\): 2\/2/); + }); + + it('mentions if: always() in the missing-shard message', () => { + vi.mocked(readShardReports).mockReturnValue([{ dir: 'a', report: shardReport(2) }]); + expect(() => runMerge({ dir: '.twd/shards' })).toThrow(/if: always\(\)/); + }); + + it('writes the merged report to the default path', () => { + vi.mocked(readShardReports).mockReturnValue([ + { dir: 'a', report: shardReport(1) }, + { dir: 'b', report: shardReport(2) }, + ]); + + expect(runMerge({ dir: '.twd/shards' })).toBe(false); + + const merged = writtenFiles().find((f) => f.endsWith('merged-run.json')); + expect(merged).toBeDefined(); + }); + + it('honors --out', () => { + vi.mocked(readShardReports).mockReturnValue([ + { dir: 'a', report: shardReport(1) }, + { dir: 'b', report: shardReport(2) }, + ]); + + runMerge({ dir: '.twd/shards', out: 'custom.json' }); + + expect(writtenFiles().some((f) => f.endsWith('custom.json'))).toBe(true); + }); + + it('returns true when any shard had a failing test', () => { + vi.mocked(readShardReports).mockReturnValue([ + { dir: 'a', report: shardReport(1) }, + { dir: 'b', report: shardReport(2, { tests: [{ id: 't2', status: 'fail', error: 'boom' }], failed: 1 }) }, + ]); + + expect(runMerge({ dir: '.twd/shards' })).toBe(true); + }); + + it('merges coverage when the run is green', () => { + vi.mocked(readShardReports).mockReturnValue([ + { dir: 'a', report: shardReport(1) }, + { dir: 'b', report: shardReport(2) }, + ]); + vi.mocked(readShardCoverage).mockReturnValue({}); + + runMerge({ dir: '.twd/shards' }); + + expect(writtenFiles().some((f) => f.includes('.nyc_output'))).toBe(true); + }); + + // The user's rule, applied to the true global result rather than one shard's. + it('skips merged coverage when the run is red', () => { + vi.mocked(readShardReports).mockReturnValue([ + { dir: 'a', report: shardReport(1) }, + { dir: 'b', report: shardReport(2, { tests: [{ id: 't2', status: 'fail', error: 'boom' }], failed: 1 }) }, + ]); + vi.mocked(readShardCoverage).mockReturnValue({}); + + runMerge({ dir: '.twd/shards' }); + + expect(writtenFiles().some((f) => f.includes('.nyc_output'))).toBe(false); + }); + + it('reports how many shards contributed coverage', () => { + const log = vi.spyOn(console, 'log'); + vi.mocked(readShardReports).mockReturnValue([ + { dir: 'a', report: shardReport(1) }, + { dir: 'b', report: shardReport(2) }, + ]); + vi.mocked(readShardCoverage).mockReturnValueOnce({}).mockReturnValueOnce(null); + + runMerge({ dir: '.twd/shards' }); + + expect(log.mock.calls.flat().join('\n')).toMatch(/Coverage merged from 1\/2 shards/); + }); + + it('says so when no shard had coverage', () => { + const log = vi.spyOn(console, 'log'); + vi.mocked(readShardReports).mockReturnValue([ + { dir: 'a', report: shardReport(1) }, + { dir: 'b', report: shardReport(2) }, + ]); + + runMerge({ dir: '.twd/shards' }); + + expect(log.mock.calls.flat().join('\n')).toMatch(/No coverage data found/); + }); + + it('returns true when contracts report an error-mode violation', () => { + const withContracts = (i) => ({ + ...shardReport(i), + contracts: { configured: true, partial: false, results: [{ alias: 'a' }], skipped: [] }, + }); + vi.mocked(readShardReports).mockReturnValue([ + { dir: 'a', report: withContracts(1) }, + { dir: 'b', report: withContracts(2) }, + ]); + vi.mocked(printContractReport).mockReturnValue(true); + + expect(runMerge({ dir: '.twd/shards' })).toBe(true); + }); + + it('warns when contract data is partial', () => { + const warn = vi.spyOn(console, 'warn'); + const partial = (i, isPartial) => ({ + ...shardReport(i), + contracts: { configured: true, partial: isPartial, results: [], skipped: [] }, + }); + vi.mocked(readShardReports).mockReturnValue([ + { dir: 'a', report: partial(1, false) }, + { dir: 'b', report: partial(2, true) }, + ]); + vi.mocked(printContractReport).mockReturnValue(false); + + runMerge({ dir: '.twd/shards' }); + + expect(warn.mock.calls.flat().join('\n')).toMatch(/contract data is partial/i); + }); + + it('prints the merged run-complete block with a shard breakdown', () => { + const log = vi.spyOn(console, 'log'); + vi.mocked(readShardReports).mockReturnValue([ + { dir: 'a', report: shardReport(1) }, + { dir: 'b', report: shardReport(2) }, + ]); + + runMerge({ dir: '.twd/shards' }); + + const output = log.mock.calls.flat().join('\n'); + expect(output).toContain('--- Run complete ---'); + expect(output).toContain('Shards: 1 ✓1 | 2 ✓1'); + }); +}); +``` + +- [ ] **Step 2: Run the test to verify it fails** + +Run: `npx vitest --run tests/mergeCommand.test.js` +Expected: FAIL — `Failed to load ../src/mergeCommand.js`. + +- [ ] **Step 3: Write the implementation** + +Create `src/mergeCommand.js`: + +```js +import fs from 'fs'; +import path from 'path'; +import { loadConfig } from './config.js'; +import { + readShardReports, + readShardCoverage, + DEFAULT_MERGED_OUT, +} from './reportFiles.js'; +import { + mergeRunReports, + findMissingShards, + reportTimings, + reportTotals, +} from './mergeReports.js'; +import { mergeCoverage } from './mergeCoverage.js'; +import { formatRunComplete } from './testSummary.js'; +import { printContractReport } from './contractReport.js'; +import { generateContractMarkdown } from './contractMarkdown.js'; + +/** + * Joins per-shard reports into one and reports on the whole run. + * + * This function owns the run's exit code. Shard jobs each exit 1 on their own + * failures, so the workflow only reaches here with `if: !cancelled()`, and the + * merged verdict is the one that counts. + */ +export function runMerge({ dir, out = null } = {}) { + if (!dir) { + throw new Error('Usage: twd-cli merge [--out ]'); + } + + const config = loadConfig(); + const workingDir = process.cwd(); + + const found = readShardReports(dir); + if (found.length === 0) { + throw new Error( + `No shard reports found in ${dir}. ` + + 'Expected /*/run.json (the actions/download-artifact layout) or /run.json.' + ); + } + + const merged = mergeRunReports(found.map((f) => f.report)); + + // Completeness is enforced here rather than inside mergeRunReports, which must + // stay associative. A gap is never a warning: a silent 3-of-4 merge reads as a + // complete green run. + const missing = findMissingShards(merged); + if (missing.length > 0) { + const total = merged.shards[0].total; + throw new Error( + `Missing shard report(s): ${missing.map((i) => `${i}/${total}`).join(', ')}. ` + + 'A shard job likely failed before uploading its artifact — check that the ' + + 'upload step runs with `if: always()`.' + ); + } + + const outPath = path.resolve(workingDir, out ?? DEFAULT_MERGED_OUT); + fs.mkdirSync(path.dirname(outPath), { recursive: true }); + fs.writeFileSync(outPath, `${JSON.stringify(merged, null, 2)}\n`); + console.log(`Merged report written to ${outPath}`); + + let hasFailures = merged.tests.some((test) => test.status === 'fail'); + + if (merged.contracts.configured) { + const validationOutput = { + results: merged.contracts.results, + skipped: merged.contracts.skipped, + }; + if (printContractReport(validationOutput)) { + hasFailures = true; + } + if (merged.contracts.partial) { + console.warn( + 'Warning: contract data is partial — at least one shard stopped early, so ' + + 'some mocks were never collected.' + ); + } + if (config.contractReportPath) { + const reportPath = path.resolve(workingDir, config.contractReportPath); + fs.mkdirSync(path.dirname(reportPath), { recursive: true }); + fs.writeFileSync(reportPath, generateContractMarkdown(validationOutput)); + console.log(`Contract report written to ${config.contractReportPath}`); + } + } + + // A red run yields no coverage — the same policy a single run has always had, + // but keyed on the whole merged result instead of one shard's. + if (config.coverage) { + const coverages = found.map((f) => + readShardCoverage(f.dir, f.report.shards[0]?.coverageFile ?? null) + ); + const contributors = coverages.filter(Boolean).length; + + if (contributors === 0) { + console.log('No coverage data found in any shard.'); + } else if (hasFailures) { + console.log( + `Skipping merged coverage — the run has failures ` + + `(${contributors}/${found.length} shard(s) had data).` + ); + } else { + const nycDir = path.resolve(workingDir, config.nycOutputDir); + fs.mkdirSync(nycDir, { recursive: true }); + fs.writeFileSync(path.join(nycDir, 'out.json'), JSON.stringify(mergeCoverage(coverages))); + console.log( + `Coverage merged from ${contributors}/${found.length} shards to ` + + `${config.nycOutputDir}/out.json` + ); + } + } + + const totals = reportTotals(merged); + if (!totals.consistent) { + console.warn( + `Warning: shard totals do not add up — ${totals.executed} executed + ` + + `${totals.notRun} not run != ${merged.discovery.totalTests} discovered. ` + + 'This points at a shard-slicing bug, not at your tests.' + ); + } + + const timings = reportTimings(merged); + console.log(''); + console.log(formatRunComplete({ + testStatus: merged.tests, + handlers: merged.handlers, + durationMs: timings.wallMs, + computeMs: timings.computeMs, + notRun: totals.notRun, + shards: merged.shards, + stoppedEarly: merged.shards.some((s) => s.stoppedEarly), + maxFailures: config.maxFailures, + })); + + return hasFailures; +} +``` + +- [ ] **Step 4: Run the test to verify it passes** + +Run: `npx vitest --run tests/mergeCommand.test.js` +Expected: PASS, 15 tests. + +- [ ] **Step 5: Wire the subcommand into the CLI** + +In `bin/twd-cli.js`, extend the imports: + +```js +import { runTests } from '../src/index.js'; +import { parseRunArgs, parseMergeArgs } from '../src/parseArgs.js'; +import { runMerge } from '../src/mergeCommand.js'; +``` + +Change the `run` branch to forward the new options: + +```js + const { testFilters, record, shard, reportDir } = parseRunArgs(process.argv.slice(3)); + const hasFailures = await runTests({ + testFilters, + recordOverrides: record, + shard, + reportDir, + }); +``` + +Add a `merge` branch immediately after the `run` block's closing brace: + +```js +} else if (command === 'merge') { + try { + const { dir, out } = parseMergeArgs(process.argv.slice(3)); + const hasFailures = runMerge({ dir, out }); + process.exit(hasFailures ? 1 : 0); + } catch (error) { + if (!error?.reported) { + console.error(error?.message ?? String(error)); + } + process.exit(1); + } +} else { +``` + +- [ ] **Step 6: Update the help text** + +In the same file's help block, add to `Usage:`: + +``` + npx twd-cli run --shard 2/4 Run only this shard's slice of the suite + and write a report to ./.twd/run + npx twd-cli merge Merge shard reports from into one + report, and exit 1 if the whole run failed +``` + +Add to `Options:`: + +``` + --shard / Run slice i of n. Each shard discovers the whole + suite and takes every nth test, so the test count + never has to be known in advance. Implies a report. + --report-dir Where to write the shard report (default ./.twd/run) +``` + +And add an example: + +``` + npx twd-cli run --shard 2/4 + npx twd-cli merge .twd/shards +``` + +- [ ] **Step 7: Verify the CLI end to end by hand** + +```bash +node ./bin/twd-cli.js merge +``` + +Expected: prints `Usage: twd-cli merge [--out ]` and exits 1. + +```bash +node ./bin/twd-cli.js merge /tmp/definitely-not-here; echo "exit=$?" +``` + +Expected: prints `No shard reports found in /tmp/definitely-not-here. ...` and `exit=1`. + +```bash +node ./bin/twd-cli.js +``` + +Expected: help text including the `--shard` and `merge` entries. + +- [ ] **Step 8: Run the full suite and commit** + +Run: `npm run test:ci` +Expected: PASS. + +```bash +git add src/mergeCommand.js tests/mergeCommand.test.js bin/twd-cli.js +git commit -m "feat(cli): add the merge command and wire shard flags through bin" +``` + +--- + +### Task 10: End-to-end verification of the CI plumbing + +**Files:** +- Modify: `.github/workflows/e2e.yml` + +**Interfaces:** +- Consumes: the finished CLI from Tasks 1-9. +- Produces: nothing consumed by later tasks. + +Unit tests cannot catch a missing `if: always()`, a wrong artifact path, or a `fail-fast` that cancels siblings. This job is the only place the real plumbing runs. + +- [ ] **Step 1: Add the sharded job** + +Append to `.github/workflows/e2e.yml`, after the existing `e2e` job (same indentation level — two spaces, a sibling of `unit-tests` and `e2e`): + +```yaml + e2e-sharded: + runs-on: ubuntu-latest + + strategy: + # Without this, the first red shard cancels its siblings and the merge job + # sees gaps it cannot distinguish from a crashed shard. + fail-fast: false + matrix: + shard: [1, 2] + + steps: + - name: Checkout repo + uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd # v5 + + - name: Setup Node.js + uses: actions/setup-node@a0853c24544627f65ddf259abe73b1d18a591444 # v5 + with: + node-version: 24 + cache: npm + + - name: Install CLI dependencies + run: npm ci + + - name: Install test-example-app dependencies + working-directory: test-example-app + run: npm install + + - name: Install mock service worker + working-directory: test-example-app + run: npx twd-js init public --save + + - name: Cache Puppeteer browsers + uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4 + with: + path: ~/.cache/puppeteer + key: ${{ runner.os }}-puppeteer-${{ hashFiles('package-lock.json') }} + restore-keys: | + ${{ runner.os }}-puppeteer- + + - name: Install Chrome for Puppeteer + run: npx puppeteer browsers install chrome + + - name: Start dev server + working-directory: test-example-app + run: | + nohup npx vite --host > vite.log 2>&1 & + npx wait-on http://localhost:5173 --timeout 30000 + + - name: Run TWD tests for this shard + working-directory: test-example-app + run: node ../bin/twd-cli.js run --shard ${{ matrix.shard }}/2 + + - name: Upload shard report + # Always: a red shard must still upload, or merge cannot tell "this shard + # failed" from "this shard never ran". + if: always() + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 + with: + name: twd-run-${{ matrix.shard }} + path: test-example-app/.twd/run + if-no-files-found: error + + e2e-merge: + runs-on: ubuntu-latest + needs: [e2e-sharded] + # Runs even though a shard job may have exited 1. Without this the merged + # summary — the point of the exercise — is never printed. + if: ${{ !cancelled() }} + + steps: + - name: Checkout repo + uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd # v5 + + - name: Setup Node.js + uses: actions/setup-node@a0853c24544627f65ddf259abe73b1d18a591444 # v5 + with: + node-version: 24 + cache: npm + + - name: Install CLI dependencies + run: npm ci + + - name: Download shard reports + uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4 + with: + pattern: twd-run-* + path: test-example-app/.twd/shards + + - name: Merge shard reports + working-directory: test-example-app + run: node ../bin/twd-cli.js merge .twd/shards + + - name: Verify the merged report + working-directory: test-example-app + run: | + if [ ! -f .twd/merged-run.json ]; then + echo "ERROR: merged report not generated" + exit 1 + fi + node -e " + const r = require('./.twd/merged-run.json'); + if (r.shards.length !== 2) { + console.error('ERROR: expected 2 shards, got ' + r.shards.length); + process.exit(1); + } + if (r.tests.length !== r.discovery.totalTests) { + console.error('ERROR: ' + r.tests.length + ' merged tests but ' + + r.discovery.totalTests + ' discovered'); + process.exit(1); + } + console.log('Merged ' + r.tests.length + ' tests from ' + r.shards.length + ' shards'); + " +``` + +- [ ] **Step 2: Verify the action SHAs resolve** + +The `upload-artifact` and `download-artifact` SHAs above must be real v4 tags. Confirm before pushing: + +```bash +gh api repos/actions/upload-artifact/git/ref/tags/v4 --jq .object.sha +gh api repos/actions/download-artifact/git/ref/tags/v4 --jq .object.sha +``` + +Replace the pinned SHAs in the YAML with whatever these print, keeping the `# v4` comment. Every other action in this file is SHA-pinned; these must match that convention. + +- [ ] **Step 3: Validate the YAML parses** + +```bash +node -e " + const fs = require('fs'); + const text = fs.readFileSync('.github/workflows/e2e.yml', 'utf-8'); + if (!text.includes('e2e-sharded') || !text.includes('e2e-merge')) { + throw new Error('jobs missing'); + } + console.log('jobs present'); +" +npx --yes yaml-lint .github/workflows/e2e.yml 2>/dev/null || echo "(yaml-lint unavailable — rely on CI)" +``` + +- [ ] **Step 4: Commit** + +```bash +git add .github/workflows/e2e.yml +git commit -m "ci: verify sharded runs and merge end to end" +``` + +- [ ] **Step 5: Push and confirm the workflow is green** + +```bash +git push -u origin feat/shardable-run-artifacts +gh run watch +``` + +Expected: `unit-tests`, `e2e`, both `e2e-sharded` matrix legs, and `e2e-merge` all pass. If `e2e-merge` reports a missing shard, the upload path or artifact name is wrong — not the merge logic. + +--- + +### Task 11: Documentation and the beta version bump + +**Files:** +- Modify: `README.md` (new section after "CI/CD Integration", before "Contract Validation" at line 270) +- Modify: `CHANGELOG.md` +- Modify: `package.json`, `package-lock.json` + +**Interfaces:** +- Consumes: everything above. +- Produces: a publishable `1.5.0-beta.0`. + +- [ ] **Step 1: Document sharding in the README** + +Insert a `## Sharding across CI jobs` section before `## Contract Validation` (currently line 270): + +````markdown +## Sharding across CI jobs + +A single run walks the whole suite in one browser. `--shard` splits it across +parallel CI jobs instead. + +```bash +npx twd-cli run --shard 2/4 # "I am job 2 of 4" +npx twd-cli merge .twd/shards # join the reports back together +``` + +The `4` is how many jobs you are running, **not** how many tests exist. You never +need to know the test count: each shard boots its own browser, discovers the whole +suite exactly as a normal run does, and keeps every 4th test. Add tests and the +same 4 jobs just split more of them. + +Each shard writes `run.json` and `coverage.json` to `./.twd/run` (change it with +`--report-dir`). `merge` reads the downloaded shard directories, combines the test +results, coverage and contract validation, prints one summary, and exits non-zero +if anything failed anywhere. + +```yaml +jobs: + test: + strategy: + fail-fast: false # or one red shard cancels the rest + matrix: + shard: [1, 2, 3, 4] + steps: + # ...checkout, npm ci, chrome, dev server... + - run: npx twd-cli run --shard ${{ matrix.shard }}/4 + - uses: actions/upload-artifact@v4 + if: always() # a red shard must still upload + with: + name: twd-run-${{ matrix.shard }} + path: .twd/run + + merge: + needs: [test] + if: ${{ !cancelled() }} # runs even though a shard went red + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: 24 + - run: npm ci + - uses: actions/download-artifact@v4 + with: + pattern: twd-run-* + path: .twd/shards + - run: npx twd-cli merge .twd/shards +``` + +Those three conditions are easy to miss and each one breaks the run: +`fail-fast: false` stops a red shard cancelling its siblings, `if: always()` on +upload keeps a red shard's report, and `if: ${{ !cancelled() }}` on merge lets the +summary print at all. + +### Notes + +- **Coverage.** Each shard writes its own `coverage.json`; `merge` combines them + into `.nyc_output/out.json` — but only when the whole run is green, matching how + a single run behaves. `merge` reports how many shards contributed. +- **Missing shards are an error.** If a shard job dies before uploading, `merge` + refuses and names the gap rather than silently reporting 3 of 4 shards as a + complete green run. +- **Tests must register identically in every job.** Each shard fingerprints the + test list it discovered and `merge` verifies they match. Registering tests + conditionally — behind a feature flag, a date, `Math.random()` — makes the + fingerprints diverge and `merge` will say so. +- **`maxFailures` is per shard.** Four shards at the default of 10 can reach 40 + failures between them before all four bail. +- **`--test` and `--shard` compose:** filters resolve first, then the filtered list + is sharded. As with any filtered run, coverage is skipped. +- **Recording** produces one clip per shard; they are not concatenated. +```` + +- [ ] **Step 2: Add the CHANGELOG entry** + +Prepend to `CHANGELOG.md`, matching the existing `## version (date)` format: + +```markdown +## 1.5.0-beta.0 (2026-08-19) + +* feat(shard): `--shard /` runs one slice of the suite so a run can be split across parallel CI jobs. Each shard discovers the whole suite itself and takes every nth test, so the test count never has to be known in advance +* feat(shard): a sharded run writes `run.json` and `coverage.json` to `./.twd/run` (`--report-dir` to change it) — the first machine-readable output twd-cli has had +* feat(merge): `npx twd-cli merge ` joins shard reports into one report covering test results, coverage and contract validation, prints a single summary with a per-shard breakdown, and owns the exit code +* feat(merge): a missing shard report is an error naming the gap, not a silently incomplete report. Shards also fingerprint the test list they discovered, so shards that saw different test sets refuse to merge +* note: no behavior change without `--shard`. A plain run writes the same files, prints the same output, and exits the same way as 1.4.0 + +Sharding needs three things right in the workflow: `fail-fast: false` on the +matrix, `if: always()` on the shard's artifact upload, and +`if: ${{ !cancelled() }}` on the merge job. See "Sharding across CI jobs" in the +README. + +This is a prerelease, published under the `beta` dist-tag: +`npm install twd-cli@beta`. +``` + +- [ ] **Step 3: Bump the version** + +```bash +npm pkg set version=1.5.0-beta.0 +node -p "require('./package.json').version" +``` + +Expected: `1.5.0-beta.0` + +- [ ] **Step 4: Regenerate the lockfile** + +```bash +npm run lock:linux +``` + +`package-lock.json` carries the version in **two** places — the top-level +`version` and `packages[""].version`. Confirm both moved: + +```bash +node -e " + const lock = require('./package-lock.json'); + const root = lock.packages[''].version; + console.log('top-level:', lock.version, '| packages[\"\"]:', root); + if (lock.version !== '1.5.0-beta.0' || root !== '1.5.0-beta.0') { + throw new Error('lockfile version fields disagree with package.json'); + } +" +``` + +- [ ] **Step 5: Verify the package contents** + +```bash +npm pack --dry-run +``` + +Expected: `bin/`, `src/` (including the five new modules), `README.md`, `LICENSE`. No `tests/`, no `test-example-app/`, no `.twd/`. + +- [ ] **Step 6: Run the full suite one last time** + +```bash +npm run test:ci +``` + +Expected: PASS with no coverage regression on `src/**`. + +- [ ] **Step 7: Commit and push** + +```bash +git add README.md CHANGELOG.md package.json package-lock.json +git commit -m "chore(release): 1.5.0-beta.0" +git push +``` + +- [ ] **Step 8: Hand back for the release** + +The version bump normally happens on `main`, but it lives on this branch by +explicit request so the beta can be tested before merging. Do **not** create the +GitHub Release from this branch. Report to the user that the branch is ready, and +that publishing means: + +1. Merge `feat/shardable-run-artifacts` into `main`. +2. Create a GitHub Release tagged `v1.5.0-beta.0`, **marked as a prerelease**. +3. `publish.yml` sees `prerelease == true` and publishes with `--tag beta`, so + `npm install twd-cli` keeps resolving to 1.4.0. + +--- + +## Verification Checklist + +Run after all tasks are complete. + +- [ ] `npm run test:ci` passes. +- [ ] `node ./bin/twd-cli.js` prints help including `--shard`, `--report-dir` and `merge`. +- [ ] `node ./bin/twd-cli.js merge` exits 1 with the usage message. +- [ ] `node ./bin/twd-cli.js run --shard 5/4` exits 1 with `Invalid --shard`. +- [ ] In `test-example-app` with a dev server running: `node ../bin/twd-cli.js run --shard 1/2` then `--shard 2/2` (moving `.twd/run` to `.twd/shards/a` and `.twd/shards/b` between runs), then `node ../bin/twd-cli.js merge .twd/shards` prints a `Shards: 1 ✓… | 2 ✓…` line and a test count equal to a full unsharded run. +- [ ] Deleting one shard directory and re-running `merge` errors with `Missing shard report(s)`. +- [ ] `node ../bin/twd-cli.js run` with no flags produces byte-identical output to 1.4.0 (`git stash` the branch and compare). +- [ ] CI green on all five jobs. From 976c37279d95d3433af8808f7b20b11e7ed56f30 Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Wed, 19 Aug 2026 21:44:04 +0200 Subject: [PATCH 06/26] docs(plan): share readValue between both arg parsers parseMergeArgs duplicated parseRunArgs's readValue closure verbatim, which violates the plan's own DRY constraint. Hoisted to module scope instead. Co-Authored-By: Claude Opus 5 (1M context) --- .../2026-08-19-shardable-run-artifacts.md | 33 +++++++++++++------ 1 file changed, 23 insertions(+), 10 deletions(-) diff --git a/docs/superpowers/plans/2026-08-19-shardable-run-artifacts.md b/docs/superpowers/plans/2026-08-19-shardable-run-artifacts.md index 2a42ec8..2f3597f 100644 --- a/docs/superpowers/plans/2026-08-19-shardable-run-artifacts.md +++ b/docs/superpowers/plans/2026-08-19-shardable-run-artifacts.md @@ -325,6 +325,26 @@ In `src/parseArgs.js`, add the import at the top: import { parseShardSpec } from './shard.js'; ``` +Hoist the existing `readValue` closure to module scope so `parseMergeArgs` can +share it rather than duplicating it. Delete the `const readValue = ...` block +from inside `parseRunArgs` and add above it: + +```js +// Reads a flag's value in either `--flag value` or `--flag=value` form, and +// reports how many tokens it consumed. Shared by both parsers. +function readValue(argv, token, prefix, index) { + if (token === prefix) { + return { value: argv[index + 1], consumed: argv[index + 1] !== undefined ? 2 : 1 }; + } + return { value: token.slice(prefix.length + 1), consumed: 1 }; +} +``` + +Then update every call site inside `parseRunArgs` to pass `argv` first — there +are four existing ones (`--test`, `--record-dir`, `--record-speed`, +`--record-pace`), each becoming e.g. `readValue(argv, token, '--test', i)`. The +17 existing tests in `tests/parseArgs.test.js` guard this refactor. + Inside `parseRunArgs`, add two declarations next to the existing ones: ```js @@ -339,13 +359,13 @@ Add two branches to the token loop, after the `--test` branch: ```js } else if (token === '--shard' || token.startsWith('--shard=')) { - const { value, consumed } = readValue(token, '--shard', i); + const { value, consumed } = readValue(argv, token, '--shard', i); // Throws on a malformed spec. A silently-ignored --shard would run zero // tests and exit 0. shard = parseShardSpec(value); i += consumed - 1; } else if (token === '--report-dir' || token.startsWith('--report-dir=')) { - const { value, consumed } = readValue(token, '--report-dir', i); + const { value, consumed } = readValue(argv, token, '--report-dir', i); if (value !== undefined) reportDir = value; i += consumed - 1; ``` @@ -366,18 +386,11 @@ export function parseMergeArgs(argv) { let dir = null; let out = null; - const readValue = (token, prefix, index) => { - if (token === prefix) { - return { value: argv[index + 1], consumed: argv[index + 1] !== undefined ? 2 : 1 }; - } - return { value: token.slice(prefix.length + 1), consumed: 1 }; - }; - for (let i = 0; i < argv.length; i++) { const token = argv[i]; if (token === '--out' || token.startsWith('--out=')) { - const { value, consumed } = readValue(token, '--out', i); + const { value, consumed } = readValue(argv, token, '--out', i); if (value !== undefined) out = value; i += consumed - 1; } else if (!token.startsWith('--') && dir === null) { From 10d35cc11bad37c9da903426a966dc84bad887ca Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Wed, 19 Aug 2026 21:45:32 +0200 Subject: [PATCH 07/26] feat(shard): parse shard specs and slice test ids round-robin --- src/shard.js | 38 +++++++++++++++++++++++ tests/shard.test.js | 73 +++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 111 insertions(+) create mode 100644 src/shard.js create mode 100644 tests/shard.test.js diff --git a/src/shard.js b/src/shard.js new file mode 100644 index 0000000..81447ac --- /dev/null +++ b/src/shard.js @@ -0,0 +1,38 @@ +// Parses a "/" shard spec, e.g. "2/4" for job 2 of 4. +// +// This throws where src/parseArgs.js silently ignores a malformed +// --record-speed, and the divergence is deliberate: "--shard 5/4" would select +// no tests and exit 0, which reads as a green build that tested nothing. +export function parseShardSpec(value) { + const raw = typeof value === 'string' ? value.trim() : value; + const match = /^(\d+)\/(\d+)$/.exec(String(raw ?? '')); + if (!match) { + throw new Error( + `Invalid --shard "${value}". Expected /, e.g. --shard 2/4.` + ); + } + + const index = Number(match[1]); + const total = Number(match[2]); + + if (total < 1) { + throw new Error(`Invalid --shard "${value}". Total must be at least 1.`); + } + if (index < 1 || index > total) { + throw new Error( + `Invalid --shard "${value}". Index must be between 1 and ${total}.` + ); + } + + return { index, total }; +} + +// Round-robin slice of an ordered id list. `index` is 1-based. +// +// Round-robin rather than contiguous: it balances better when adjacent tests +// have similar cost, and nothing in twd-js requires a suite to run +// contiguously, so locality buys nothing. An empty result is a legal outcome — +// 3 tests across 4 shards leaves the fourth with nothing to run. +export function selectShardIds(ids, index, total) { + return ids.filter((_, i) => i % total === index - 1); +} diff --git a/tests/shard.test.js b/tests/shard.test.js new file mode 100644 index 0000000..b3e5bd4 --- /dev/null +++ b/tests/shard.test.js @@ -0,0 +1,73 @@ +import { describe, it, expect } from 'vitest'; +import { parseShardSpec, selectShardIds } from '../src/shard.js'; + +describe('parseShardSpec', () => { + it('parses /', () => { + expect(parseShardSpec('2/4')).toEqual({ index: 2, total: 4 }); + expect(parseShardSpec('1/1')).toEqual({ index: 1, total: 1 }); + expect(parseShardSpec(' 3/4 ')).toEqual({ index: 3, total: 4 }); + }); + + // Unlike --record-speed, a bad --shard must never be silently ignored: it + // would run zero tests and exit 0, reading as a green build that tested + // nothing. + it('throws when the index exceeds the total', () => { + expect(() => parseShardSpec('5/4')).toThrow(/between 1 and 4/); + }); + + it('throws on a zero or negative index', () => { + expect(() => parseShardSpec('0/4')).toThrow(/between 1 and 4/); + expect(() => parseShardSpec('-1/4')).toThrow(/Expected \//); + }); + + it('throws on a zero total', () => { + expect(() => parseShardSpec('2/0')).toThrow(/at least 1/); + }); + + it('throws on unparseable input', () => { + for (const bad of ['abc', '', '2', '2/', '/4', '2.5/4', '2/4/6', undefined, null]) { + expect(() => parseShardSpec(bad)).toThrow(/Expected \//); + } + }); + + it('names the offending value in the message', () => { + expect(() => parseShardSpec('9/2')).toThrow(/"9\/2"/); + }); +}); + +describe('selectShardIds', () => { + const ids = Array.from({ length: 12 }, (_, i) => `t${i}`); + + it('takes every Nth id at its own offset', () => { + expect(selectShardIds(ids, 1, 4)).toEqual(['t0', 't4', 't8']); + expect(selectShardIds(ids, 2, 4)).toEqual(['t1', 't5', 't9']); + expect(selectShardIds(ids, 4, 4)).toEqual(['t3', 't7', 't11']); + }); + + it('returns everything when total is 1', () => { + expect(selectShardIds(ids, 1, 1)).toEqual(ids); + }); + + // 3 tests across 4 shards leaves the fourth with nothing. Legal, not an error. + it('returns an empty slice when there are fewer ids than shards', () => { + expect(selectShardIds(['a', 'b', 'c'], 4, 4)).toEqual([]); + expect(selectShardIds([], 1, 4)).toEqual([]); + }); + + // The property that makes sharding trustworthy: nothing lost, nothing doubled. + it('partitions the input — every id lands in exactly one shard', () => { + const many = Array.from({ length: 37 }, (_, i) => `t${i}`); + const total = 5; + const slices = Array.from({ length: total }, (_, i) => selectShardIds(many, i + 1, total)); + const counts = new Map(); + for (const id of slices.flat()) counts.set(id, (counts.get(id) ?? 0) + 1); + expect(counts.size).toBe(many.length); + expect([...counts.values()]).toEqual(many.map(() => 1)); + }); + + it('does not mutate its input', () => { + const input = ['a', 'b', 'c']; + selectShardIds(input, 1, 2); + expect(input).toEqual(['a', 'b', 'c']); + }); +}); From 995fa8a9b92d1d5543335cd302287f681ee0e4c8 Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Wed, 19 Aug 2026 21:50:46 +0200 Subject: [PATCH 08/26] feat(cli): add --shard, --report-dir, and merge arg parsing --- src/parseArgs.js | 61 ++++++++++++++++++++++++++------- tests/parseArgs.test.js | 74 +++++++++++++++++++++++++++++++++++++++-- 2 files changed, 120 insertions(+), 15 deletions(-) diff --git a/src/parseArgs.js b/src/parseArgs.js index 75f9965..873897f 100644 --- a/src/parseArgs.js +++ b/src/parseArgs.js @@ -1,36 +1,52 @@ +import { parseShardSpec } from './shard.js'; + +// Reads a flag's value in either `--flag value` or `--flag=value` form, and +// reports how many tokens it consumed. Shared by both parsers. +function readValue(argv, token, prefix, index) { + if (token === prefix) { + return { value: argv[index + 1], consumed: argv[index + 1] !== undefined ? 2 : 1 }; + } + return { value: token.slice(prefix.length + 1), consumed: 1 }; +} + export function parseRunArgs(argv) { const testFilters = []; const record = {}; - - const readValue = (token, prefix, index) => { - if (token === prefix) { - return { value: argv[index + 1], consumed: argv[index + 1] !== undefined ? 2 : 1 }; - } - return { value: token.slice(prefix.length + 1), consumed: 1 }; - }; + let shard = null; + let reportDir = null; for (let i = 0; i < argv.length; i++) { const token = argv[i]; if (token === '--test' || token.startsWith('--test=')) { - const { value, consumed } = readValue(token, '--test', i); + const { value, consumed } = readValue(argv, token, '--test', i); if (value !== undefined) testFilters.push(value); i += consumed - 1; + } else if (token === '--shard' || token.startsWith('--shard=')) { + const { value, consumed } = readValue(argv, token, '--shard', i); + // Throws on a malformed spec. A silently-ignored --shard would run zero + // tests and exit 0. + shard = parseShardSpec(value); + i += consumed - 1; + } else if (token === '--report-dir' || token.startsWith('--report-dir=')) { + const { value, consumed } = readValue(argv, token, '--report-dir', i); + if (value !== undefined) reportDir = value; + i += consumed - 1; } else if (token === '--record') { record.enabled = true; } else if (token === '--record-dir' || token.startsWith('--record-dir=')) { - const { value, consumed } = readValue(token, '--record-dir', i); + const { value, consumed } = readValue(argv, token, '--record-dir', i); if (value !== undefined) record.dir = value; i += consumed - 1; } else if (token === '--record-speed' || token.startsWith('--record-speed=')) { - const { value, consumed } = readValue(token, '--record-speed', i); + const { value, consumed } = readValue(argv, token, '--record-speed', i); const parsed = Number(value); if (value !== undefined && Number.isFinite(parsed) && parsed > 0) { record.speed = parsed; } i += consumed - 1; } else if (token === '--record-pace' || token.startsWith('--record-pace=')) { - const { value, consumed } = readValue(token, '--record-pace', i); + const { value, consumed } = readValue(argv, token, '--record-pace', i); const parsed = Number(value); if (value !== undefined && Number.isFinite(parsed) && parsed > 0) { record.pace = parsed; @@ -39,5 +55,26 @@ export function parseRunArgs(argv) { } } - return { testFilters, record }; + return { testFilters, record, shard, reportDir }; +} + +// `twd-cli merge [--out ]`. The directory is the first positional +// token; anything after the first is ignored. +export function parseMergeArgs(argv) { + let dir = null; + let out = null; + + for (let i = 0; i < argv.length; i++) { + const token = argv[i]; + + if (token === '--out' || token.startsWith('--out=')) { + const { value, consumed } = readValue(argv, token, '--out', i); + if (value !== undefined) out = value; + i += consumed - 1; + } else if (!token.startsWith('--') && dir === null) { + dir = token; + } + } + + return { dir, out }; } diff --git a/tests/parseArgs.test.js b/tests/parseArgs.test.js index b2eb066..d9103a4 100644 --- a/tests/parseArgs.test.js +++ b/tests/parseArgs.test.js @@ -1,15 +1,17 @@ import { describe, it, expect } from "vitest"; -import { parseRunArgs } from "../src/parseArgs.js"; +import { parseRunArgs, parseMergeArgs } from "../src/parseArgs.js"; describe("parseRunArgs", () => { it("returns empty filters when no args", () => { - expect(parseRunArgs([])).toEqual({ testFilters: [], record: {} }); + expect(parseRunArgs([])).toEqual({ testFilters: [], record: {}, shard: null, reportDir: null }); }); it("parses a single --test ", () => { expect(parseRunArgs(['--test', 'shows error'])).toEqual({ testFilters: ['shows error'], record: {}, + shard: null, + reportDir: null, }); }); @@ -17,6 +19,8 @@ describe("parseRunArgs", () => { expect(parseRunArgs(['--test', 'Login', '--test', 'Signup'])).toEqual({ testFilters: ['Login', 'Signup'], record: {}, + shard: null, + reportDir: null, }); }); @@ -24,17 +28,21 @@ describe("parseRunArgs", () => { expect(parseRunArgs(['--test=Login'])).toEqual({ testFilters: ['Login'], record: {}, + shard: null, + reportDir: null, }); }); it("ignores a trailing --test with no value", () => { - expect(parseRunArgs(['--test'])).toEqual({ testFilters: [], record: {} }); + expect(parseRunArgs(['--test'])).toEqual({ testFilters: [], record: {}, shard: null, reportDir: null }); }); it("ignores unknown tokens", () => { expect(parseRunArgs(['--verbose', '--test', 'Login'])).toEqual({ testFilters: ['Login'], record: {}, + shard: null, + reportDir: null, }); }); @@ -71,6 +79,8 @@ describe("parseRunArgs", () => { expect(parseRunArgs(['--record', '--test', 'checkout', '--record-speed=0.5'])).toEqual({ testFilters: ['checkout'], record: { enabled: true, speed: 0.5 }, + shard: null, + reportDir: null, }); }); @@ -93,7 +103,65 @@ describe("parseRunArgs", () => { expect(parseRunArgs(['--record', '--test', 'checkout', '--record-pace=500'])).toEqual({ testFilters: ['checkout'], record: { enabled: true, pace: 500 }, + shard: null, + reportDir: null, }); }); }); + +describe('parseRunArgs shard and report flags', () => { + it('parses --shard in both forms', () => { + expect(parseRunArgs(['--shard', '2/4']).shard).toEqual({ index: 2, total: 4 }); + expect(parseRunArgs(['--shard=3/4']).shard).toEqual({ index: 3, total: 4 }); + }); + + it('throws on an invalid --shard instead of ignoring it', () => { + expect(() => parseRunArgs(['--shard', '5/4'])).toThrow(/Invalid --shard/); + expect(() => parseRunArgs(['--shard=abc'])).toThrow(/Invalid --shard/); + }); + + it('throws on a trailing --shard with no value', () => { + expect(() => parseRunArgs(['--shard'])).toThrow(/Invalid --shard/); + }); + + it('parses --report-dir in both forms', () => { + expect(parseRunArgs(['--report-dir', './out']).reportDir).toBe('./out'); + expect(parseRunArgs(['--report-dir=./out']).reportDir).toBe('./out'); + }); + + it('ignores a trailing --report-dir with no value', () => { + expect(parseRunArgs(['--report-dir']).reportDir).toBeNull(); + }); + + it('combines --shard with --test filters and record flags', () => { + expect(parseRunArgs(['--shard', '2/4', '--test', 'Login', '--record'])).toEqual({ + testFilters: ['Login'], + record: { enabled: true }, + shard: { index: 2, total: 4 }, + reportDir: null, + }); + }); +}); + +describe('parseMergeArgs', () => { + it('reads the directory as the first positional', () => { + expect(parseMergeArgs(['.twd/shards'])).toEqual({ dir: '.twd/shards', out: null }); + }); + + it('parses --out in both forms', () => { + expect(parseMergeArgs(['.twd/shards', '--out', 'merged.json'])) + .toEqual({ dir: '.twd/shards', out: 'merged.json' }); + expect(parseMergeArgs(['.twd/shards', '--out=merged.json'])) + .toEqual({ dir: '.twd/shards', out: 'merged.json' }); + }); + + it('returns a null dir when none is given', () => { + expect(parseMergeArgs([])).toEqual({ dir: null, out: null }); + expect(parseMergeArgs(['--out=merged.json'])).toEqual({ dir: null, out: 'merged.json' }); + }); + + it('takes only the first positional as the directory', () => { + expect(parseMergeArgs(['a', 'b']).dir).toBe('a'); + }); +}); From 40ae283682565873d7090c923701c0f9cb4f478d Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Wed, 19 Aug 2026 21:54:37 +0200 Subject: [PATCH 09/26] feat(report): build run reports and fingerprint discovered tests --- src/runReport.js | 83 +++++++++++++++++++++++++++ tests/runReport.test.js | 123 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 206 insertions(+) create mode 100644 src/runReport.js create mode 100644 tests/runReport.test.js diff --git a/src/runReport.js b/src/runReport.js new file mode 100644 index 0000000..ef043ec --- /dev/null +++ b/src/runReport.js @@ -0,0 +1,83 @@ +import crypto from 'node:crypto'; + +export const REPORT_SCHEMA_VERSION = 1; + +/** + * Hash of the full ordered test id list plus any active --test filters. + * + * Round-robin sharding is correct only if every job enumerates an identical + * test set in an identical order. That breaks silently if the app registers + * tests conditionally — a feature flag, a date, Math.random — or if two shard + * jobs did not build the same code: tests quietly never run and the build stays + * green. Shards compare fingerprints at merge time so it becomes an error. + * + * Filters are OR'd, so their order carries no meaning and is normalized away. + */ +export function fingerprintTests(orderedIds, filters = []) { + const payload = JSON.stringify({ + orderedIds, + filters: [...filters].sort(), + }); + const digest = crypto.createHash('sha256').update(payload).digest('hex'); + return `sha256:${digest}`; +} + +/** + * Assembles the on-disk run report. Pure: no I/O, no clock reads. + * + * `shards` is an array even for a single run, because a merged report has the + * same shape as a single-shard one. That is what makes merging associative and + * lets one set of formatters render both. + * + * `startedAt` and `endedAt` are epoch milliseconds; the report stores ISO + * strings plus the derived duration. + */ +export function buildRunReport({ + shard, + startedAt, + endedAt, + allTestIds, + filters = [], + handlers, + tests, + executed, + notRun, + stoppedEarly, + coverageFile = null, + recording = null, + contracts = null, +}) { + return { + schemaVersion: REPORT_SCHEMA_VERSION, + shards: [ + { + index: shard.index, + total: shard.total, + startedAt: new Date(startedAt).toISOString(), + endedAt: new Date(endedAt).toISOString(), + durationMs: endedAt - startedAt, + executed, + notRun, + // Merged reports do not record which shard ran a test, so the per-shard + // breakdown line could not be rendered without this count. + failed: tests.filter((t) => t.status === 'fail').length, + stoppedEarly, + coverageFile, + recording, + }, + ], + discovery: { + totalTests: allTestIds.length, + fingerprint: fingerprintTests(allTestIds, filters), + }, + selection: { filters: [...filters] }, + handlers, + tests, + contracts: contracts ?? { + configured: false, + partial: false, + results: [], + skipped: [], + }, + }; +} diff --git a/tests/runReport.test.js b/tests/runReport.test.js new file mode 100644 index 0000000..ba57eb8 --- /dev/null +++ b/tests/runReport.test.js @@ -0,0 +1,123 @@ +import { describe, it, expect } from 'vitest'; +import { buildRunReport, fingerprintTests, REPORT_SCHEMA_VERSION } from '../src/runReport.js'; + +const handlers = [ + { id: 's1', name: 'Login', parent: null, type: 'suite' }, + { id: 't1', name: 'works', parent: 's1', type: 'test' }, +]; + +function build(overrides = {}) { + return buildRunReport({ + shard: { index: 2, total: 4 }, + startedAt: 1_000, + endedAt: 4_500, + allTestIds: ['t1', 't2', 't3'], + filters: [], + handlers, + tests: [{ id: 't1', status: 'pass' }], + executed: 1, + notRun: 0, + stoppedEarly: false, + ...overrides, + }); +} + +describe('fingerprintTests', () => { + it('is stable for the same input', () => { + expect(fingerprintTests(['a', 'b'])).toBe(fingerprintTests(['a', 'b'])); + }); + + it('is prefixed with the algorithm', () => { + expect(fingerprintTests(['a'])).toMatch(/^sha256:[0-9a-f]{64}$/); + }); + + // Order matters: round-robin slicing is only correct if every shard sees the + // same list in the same order. + it('changes when the id order changes', () => { + expect(fingerprintTests(['a', 'b'])).not.toBe(fingerprintTests(['b', 'a'])); + }); + + it('changes when the id set changes', () => { + expect(fingerprintTests(['a', 'b'])).not.toBe(fingerprintTests(['a', 'b', 'c'])); + }); + + it('changes when the filters differ', () => { + expect(fingerprintTests(['a'], ['Login'])).not.toBe(fingerprintTests(['a'], ['Cart'])); + expect(fingerprintTests(['a'], [])).not.toBe(fingerprintTests(['a'], ['Login'])); + }); + + // Filters are OR'd, so their order is not meaningful and must not split + // otherwise-identical shards. + it('ignores the order the filters were given in', () => { + expect(fingerprintTests(['a'], ['Login', 'Cart'])) + .toBe(fingerprintTests(['a'], ['Cart', 'Login'])); + }); +}); + +describe('buildRunReport', () => { + it('stamps the schema version', () => { + expect(build().schemaVersion).toBe(REPORT_SCHEMA_VERSION); + }); + + it('wraps a single shard descriptor in an array', () => { + const report = build(); + expect(report.shards).toHaveLength(1); + expect(report.shards[0]).toMatchObject({ + index: 2, total: 4, executed: 1, notRun: 0, failed: 0, stoppedEarly: false, + }); + }); + + it('derives durationMs and ISO timestamps from epoch millis', () => { + const shard = build().shards[0]; + expect(shard.durationMs).toBe(3500); + expect(shard.startedAt).toBe(new Date(1_000).toISOString()); + expect(shard.endedAt).toBe(new Date(4_500).toISOString()); + }); + + it('counts this shard\'s failures', () => { + const report = build({ + tests: [ + { id: 't1', status: 'pass' }, + { id: 't2', status: 'fail', error: 'boom' }, + { id: 't3', status: 'skip' }, + ], + }); + expect(report.shards[0].failed).toBe(1); + }); + + it('records total discovered tests and the fingerprint', () => { + const report = build(); + expect(report.discovery.totalTests).toBe(3); + expect(report.discovery.fingerprint).toBe(fingerprintTests(['t1', 't2', 't3'], [])); + }); + + it('carries handlers and tests through untouched', () => { + const report = build(); + expect(report.handlers).toEqual(handlers); + expect(report.tests).toEqual([{ id: 't1', status: 'pass' }]); + }); + + it('copies the filters rather than aliasing them', () => { + const filters = ['Login']; + const report = build({ filters }); + filters.push('Cart'); + expect(report.selection.filters).toEqual(['Login']); + }); + + it('defaults contracts to an unconfigured empty block', () => { + expect(build().contracts).toEqual({ + configured: false, partial: false, results: [], skipped: [], + }); + }); + + it('passes a contracts block through when given', () => { + const contracts = { configured: true, partial: true, results: [{ alias: 'a' }], skipped: [] }; + expect(build({ contracts }).contracts).toEqual(contracts); + }); + + it('defaults coverageFile and recording to null', () => { + const shard = build().shards[0]; + expect(shard.coverageFile).toBeNull(); + expect(shard.recording).toBeNull(); + }); +}); From b4a7f2c1db8c3b31a3a9e13845243422ee6e36e4 Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Wed, 19 Aug 2026 21:59:26 +0200 Subject: [PATCH 10/26] feat(report): read and write shard report artifacts --- src/reportFiles.js | 87 +++++++++++++++++++++++++ tests/reportFiles.test.js | 134 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 221 insertions(+) create mode 100644 src/reportFiles.js create mode 100644 tests/reportFiles.test.js diff --git a/src/reportFiles.js b/src/reportFiles.js new file mode 100644 index 0000000..747eb20 --- /dev/null +++ b/src/reportFiles.js @@ -0,0 +1,87 @@ +import fs from 'fs'; +import path from 'path'; + +export const DEFAULT_REPORT_DIR = './.twd/run'; +export const DEFAULT_MERGED_OUT = './.twd/merged-run.json'; +export const RUN_REPORT_FILE = 'run.json'; +export const COVERAGE_FILE = 'coverage.json'; + +/** + * Writes one shard's report, and its coverage when it collected any. + * + * The report is pretty-printed because a human reads it when a merge complains. + * Coverage is not: it is machine input for nyc, routinely several megabytes, and + * indenting it would roughly double the artifact size for no benefit. + */ +export function writeRunReport(dir, report, coverage = null) { + fs.mkdirSync(dir, { recursive: true }); + + const reportPath = path.join(dir, RUN_REPORT_FILE); + fs.writeFileSync(reportPath, `${JSON.stringify(report, null, 2)}\n`); + + let coveragePath = null; + if (coverage) { + coveragePath = path.join(dir, COVERAGE_FILE); + fs.writeFileSync(coveragePath, JSON.stringify(coverage)); + } + + return { reportPath, coveragePath }; +} + +function readJson(file, label) { + const raw = fs.readFileSync(file, 'utf-8'); + try { + return JSON.parse(raw); + } catch (err) { + throw new Error(`Could not parse ${label} at ${file}: ${err.message}`); + } +} + +/** + * Finds every shard report under `dir`. + * + * actions/download-artifact lays each artifact out as its own directory, so the + * normal shape is `//run.json`. A bare `/run.json` is + * also accepted, which is what a local single-shard run produces. + */ +export function readShardReports(dir) { + const found = []; + + const direct = path.join(dir, RUN_REPORT_FILE); + if (fs.existsSync(direct)) { + found.push({ dir, report: readJson(direct, RUN_REPORT_FILE) }); + } + + let entries; + try { + entries = fs.readdirSync(dir, { withFileTypes: true }); + } catch { + // Missing or unreadable directory: the caller reports "no reports found", + // which is a better message than an ENOENT stack. + return found; + } + + for (const entry of entries) { + if (!entry.isDirectory()) continue; + const shardDir = path.join(dir, entry.name); + const file = path.join(shardDir, RUN_REPORT_FILE); + if (fs.existsSync(file)) { + found.push({ dir: shardDir, report: readJson(file, RUN_REPORT_FILE) }); + } + } + + return found; +} + +/** + * Reads a shard's coverage, or null when it collected none. + * + * Absence is a normal outcome, not an error: a filtered run skips coverage + * entirely. Merge reports how many shards contributed. + */ +export function readShardCoverage(shardDir, coverageFile) { + if (!coverageFile) return null; + const file = path.join(shardDir, coverageFile); + if (!fs.existsSync(file)) return null; + return readJson(file, COVERAGE_FILE); +} diff --git a/tests/reportFiles.test.js b/tests/reportFiles.test.js new file mode 100644 index 0000000..da203bd --- /dev/null +++ b/tests/reportFiles.test.js @@ -0,0 +1,134 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest'; + +vi.mock('fs'); + +import fs from 'fs'; +import { + writeRunReport, + readShardReports, + readShardCoverage, + RUN_REPORT_FILE, + COVERAGE_FILE, + DEFAULT_REPORT_DIR, +} from '../src/reportFiles.js'; + +const report = { schemaVersion: 1, tests: [] }; + +describe('writeRunReport', () => { + beforeEach(() => { + vi.clearAllMocks(); + }); + + it('creates the directory recursively', () => { + writeRunReport('.twd/run', report, null); + expect(fs.mkdirSync).toHaveBeenCalledWith('.twd/run', { recursive: true }); + }); + + it('writes pretty-printed JSON so the report is readable by eye', () => { + writeRunReport('.twd/run', report, null); + const [file, body] = vi.mocked(fs.writeFileSync).mock.calls[0]; + expect(file).toBe(`.twd/run/${RUN_REPORT_FILE}`); + expect(body).toBe(`${JSON.stringify(report, null, 2)}\n`); + }); + + it('does not write a coverage file when there is no coverage', () => { + writeRunReport('.twd/run', report, null); + expect(fs.writeFileSync).toHaveBeenCalledTimes(1); + expect(writeRunReport('.twd/run', report, null).coveragePath).toBeNull(); + }); + + // Coverage stays raw and unformatted: it is machine input for nyc, routinely + // several megabytes, and pretty-printing it would double the artifact size. + it('writes coverage compactly alongside the report', () => { + const coverage = { '/a.js': { s: { 0: 1 } } }; + const result = writeRunReport('.twd/run', report, coverage); + expect(fs.writeFileSync).toHaveBeenCalledWith( + `.twd/run/${COVERAGE_FILE}`, + JSON.stringify(coverage), + ); + expect(result.coveragePath).toBe(`.twd/run/${COVERAGE_FILE}`); + }); +}); + +describe('readShardReports', () => { + beforeEach(() => { + vi.clearAllMocks(); + }); + + // This is download-artifact's layout: one directory per artifact name. + it('reads run.json from each child directory', () => { + vi.mocked(fs.existsSync).mockImplementation((p) => String(p).endsWith(RUN_REPORT_FILE)); + vi.mocked(fs.readdirSync).mockReturnValue([ + { name: 'twd-run-1', isDirectory: () => true }, + { name: 'twd-run-2', isDirectory: () => true }, + { name: 'notes.txt', isDirectory: () => false }, + ]); + vi.mocked(fs.readFileSync).mockReturnValue(JSON.stringify(report)); + + const found = readShardReports('.twd/shards'); + + expect(found.map((f) => f.dir)).toEqual([ + '.twd/shards', + '.twd/shards/twd-run-1', + '.twd/shards/twd-run-2', + ]); + expect(found[0].report).toEqual(report); + }); + + it('skips child directories with no run.json', () => { + vi.mocked(fs.existsSync).mockImplementation((p) => String(p).includes('twd-run-1')); + vi.mocked(fs.readdirSync).mockReturnValue([ + { name: 'twd-run-1', isDirectory: () => true }, + { name: 'empty', isDirectory: () => true }, + ]); + vi.mocked(fs.readFileSync).mockReturnValue(JSON.stringify(report)); + + expect(readShardReports('.twd/shards').map((f) => f.dir)) + .toEqual(['.twd/shards/twd-run-1']); + }); + + it('returns an empty array when the directory does not exist', () => { + vi.mocked(fs.existsSync).mockReturnValue(false); + vi.mocked(fs.readdirSync).mockImplementation(() => { + throw Object.assign(new Error('ENOENT'), { code: 'ENOENT' }); + }); + expect(readShardReports('.twd/nope')).toEqual([]); + }); + + it('explains which file failed to parse', () => { + vi.mocked(fs.existsSync).mockReturnValue(true); + vi.mocked(fs.readdirSync).mockReturnValue([]); + vi.mocked(fs.readFileSync).mockReturnValue('{ not json'); + expect(() => readShardReports('.twd/shards')).toThrow(/run\.json/); + }); +}); + +describe('readShardCoverage', () => { + beforeEach(() => { + vi.clearAllMocks(); + }); + + it('reads the named coverage file from the shard directory', () => { + vi.mocked(fs.existsSync).mockReturnValue(true); + vi.mocked(fs.readFileSync).mockReturnValue('{"/a.js":{}}'); + expect(readShardCoverage('.twd/shards/twd-run-1', COVERAGE_FILE)).toEqual({ '/a.js': {} }); + }); + + it('returns null when the shard recorded no coverage file', () => { + expect(readShardCoverage('.twd/shards/twd-run-1', null)).toBeNull(); + expect(fs.existsSync).not.toHaveBeenCalled(); + }); + + // A shard whose coverage file is absent simply does not contribute; merge + // reports the contributor count rather than failing. + it('returns null when the file is missing on disk', () => { + vi.mocked(fs.existsSync).mockReturnValue(false); + expect(readShardCoverage('.twd/shards/twd-run-1', COVERAGE_FILE)).toBeNull(); + }); +}); + +describe('defaults', () => { + it('defaults the report dir to .twd/run', () => { + expect(DEFAULT_REPORT_DIR).toBe('./.twd/run'); + }); +}); From 3266a296e829222b8e8acfef75d83bd16d849256 Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Wed, 19 Aug 2026 22:08:00 +0200 Subject: [PATCH 11/26] feat(coverage): merge per-shard Istanbul coverage maps Promotes istanbul-lib-coverage to a runtime dependency (was only transitive via @vitest/coverage-v8) since the merge command runs in a consumer project where devDependencies aren't installed. Clones each shard's coverage object before CoverageMap.merge, since FileCoverage aliases a plain object instead of copying it, which would otherwise mutate the first shard's coverage in place once a later shard's counts for the same file are merged in. --- package-lock.json | 89 ++++++++++++++++++++----------------- package.json | 1 + src/mergeCoverage.js | 26 +++++++++++ tests/mergeCoverage.test.js | 70 +++++++++++++++++++++++++++++ 4 files changed, 145 insertions(+), 41 deletions(-) create mode 100644 src/mergeCoverage.js create mode 100644 tests/mergeCoverage.test.js diff --git a/package-lock.json b/package-lock.json index 1dc1dbf..908160f 100644 --- a/package-lock.json +++ b/package-lock.json @@ -9,6 +9,7 @@ "version": "1.4.0", "license": "ISC", "dependencies": { + "istanbul-lib-coverage": "^3.2.2", "openapi-mock-validator": "^0.3.0", "puppeteer": "^25.3.0" }, @@ -137,35 +138,38 @@ } }, "node_modules/@emnapi/core": { - "version": "1.11.1", - "resolved": "https://registry.npmjs.org/@emnapi/core/-/core-1.11.1.tgz", - "integrity": "sha512-RSvbQmHzdKzNsLYa/wHrbc3KN4sYLKAdPZxqiM2HATqv/SBk2/ENSHpvXGaLOMcsAyz0poEGqkmmKYG3OWiJEQ==", + "version": "1.11.3", + "resolved": "https://registry.npmjs.org/@emnapi/core/-/core-1.11.3.tgz", + "integrity": "sha512-zLpS5asjEb7lq8jYLq37N6XKaE41DIexlY1rF/z4/tIl3wo13Sqm28fRyfIsKZD+NZ8mM5RoKkpW/rBcuoSZSg==", "dev": true, "license": "MIT", "optional": true, + "peer": true, "dependencies": { - "@emnapi/wasi-threads": "1.2.2", + "@emnapi/wasi-threads": "1.2.3", "tslib": "^2.4.0" } }, "node_modules/@emnapi/runtime": { - "version": "1.11.1", - "resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.11.1.tgz", - "integrity": "sha512-vgj7R3y3Wgx24IQaGPA/R6YFXLHVMOZ0uVEyIQPaWs+rd1AzfEMXlAC22FYwO1XkKR6NPsq7mUandH8oIRdZFw==", + "version": "1.11.3", + "resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.11.3.tgz", + "integrity": "sha512-Xz4Tpyki7XyrpbUK1jR1AhdAdaXyhhY4lZ3neLodmhpuWfy2PAQN5B46sAiU4liOXGLkHypn/qU+jvfWSCYYLA==", "dev": true, "license": "MIT", "optional": true, + "peer": true, "dependencies": { "tslib": "^2.4.0" } }, "node_modules/@emnapi/wasi-threads": { - "version": "1.2.2", - "resolved": "https://registry.npmjs.org/@emnapi/wasi-threads/-/wasi-threads-1.2.2.tgz", - "integrity": "sha512-c95qOXkHdydNKhscBTebqEC1CVAZpyqOfVfBzQ1qgzyl3gfeldUjIggDbIZgDKsHLgnsM+igH7TJ/eAasaVuMA==", + "version": "1.2.3", + "resolved": "https://registry.npmjs.org/@emnapi/wasi-threads/-/wasi-threads-1.2.3.tgz", + "integrity": "sha512-ELEBe8PsLvvJ6QMr0zLt8ffvOHW/dc1m3CEzNMg7aJUv3bMaoDtw2TXyDAwkYBuroxxuHEwhRTLJSe5sya547g==", "dev": true, "license": "MIT", "optional": true, + "peer": true, "dependencies": { "tslib": "^2.4.0" } @@ -348,9 +352,6 @@ "arm64" ], "dev": true, - "libc": [ - "glibc" - ], "license": "MIT", "optional": true, "os": [ @@ -368,9 +369,6 @@ "arm64" ], "dev": true, - "libc": [ - "musl" - ], "license": "MIT", "optional": true, "os": [ @@ -388,9 +386,6 @@ "ppc64" ], "dev": true, - "libc": [ - "glibc" - ], "license": "MIT", "optional": true, "os": [ @@ -408,9 +403,6 @@ "s390x" ], "dev": true, - "libc": [ - "glibc" - ], "license": "MIT", "optional": true, "os": [ @@ -428,9 +420,6 @@ "x64" ], "dev": true, - "libc": [ - "glibc" - ], "license": "MIT", "optional": true, "os": [ @@ -448,9 +437,6 @@ "x64" ], "dev": true, - "libc": [ - "musl" - ], "license": "MIT", "optional": true, "os": [ @@ -496,6 +482,40 @@ "node": "^20.19.0 || >=22.12.0" } }, + "node_modules/@rolldown/binding-wasm32-wasi/node_modules/@emnapi/core": { + "version": "1.11.1", + "resolved": "https://registry.npmjs.org/@emnapi/core/-/core-1.11.1.tgz", + "integrity": "sha512-RSvbQmHzdKzNsLYa/wHrbc3KN4sYLKAdPZxqiM2HATqv/SBk2/ENSHpvXGaLOMcsAyz0poEGqkmmKYG3OWiJEQ==", + "dev": true, + "license": "MIT", + "optional": true, + "dependencies": { + "@emnapi/wasi-threads": "1.2.2", + "tslib": "^2.4.0" + } + }, + "node_modules/@rolldown/binding-wasm32-wasi/node_modules/@emnapi/runtime": { + "version": "1.11.1", + "resolved": "https://registry.npmjs.org/@emnapi/runtime/-/runtime-1.11.1.tgz", + "integrity": "sha512-vgj7R3y3Wgx24IQaGPA/R6YFXLHVMOZ0uVEyIQPaWs+rd1AzfEMXlAC22FYwO1XkKR6NPsq7mUandH8oIRdZFw==", + "dev": true, + "license": "MIT", + "optional": true, + "dependencies": { + "tslib": "^2.4.0" + } + }, + "node_modules/@rolldown/binding-wasm32-wasi/node_modules/@emnapi/wasi-threads": { + "version": "1.2.2", + "resolved": "https://registry.npmjs.org/@emnapi/wasi-threads/-/wasi-threads-1.2.2.tgz", + "integrity": "sha512-c95qOXkHdydNKhscBTebqEC1CVAZpyqOfVfBzQ1qgzyl3gfeldUjIggDbIZgDKsHLgnsM+igH7TJ/eAasaVuMA==", + "dev": true, + "license": "MIT", + "optional": true, + "dependencies": { + "tslib": "^2.4.0" + } + }, "node_modules/@rolldown/binding-win32-arm64-msvc": { "version": "1.1.5", "resolved": "https://registry.npmjs.org/@rolldown/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-1.1.5.tgz", @@ -1156,7 +1176,6 @@ "version": "3.2.2", "resolved": "https://registry.npmjs.org/istanbul-lib-coverage/-/istanbul-lib-coverage-3.2.2.tgz", "integrity": "sha512-O8dpsF+r0WV/8MNRKfnmrtCWhuKjxrq2w+jpzBL5UZKTi2LeVWnWOmWRxFlesJONmc+wLAGvKQZEOanko0LFTg==", - "dev": true, "license": "BSD-3-Clause", "engines": { "node": ">=8" @@ -1369,9 +1388,6 @@ "arm64" ], "dev": true, - "libc": [ - "glibc" - ], "license": "MPL-2.0", "optional": true, "os": [ @@ -1393,9 +1409,6 @@ "arm64" ], "dev": true, - "libc": [ - "musl" - ], "license": "MPL-2.0", "optional": true, "os": [ @@ -1417,9 +1430,6 @@ "x64" ], "dev": true, - "libc": [ - "glibc" - ], "license": "MPL-2.0", "optional": true, "os": [ @@ -1441,9 +1451,6 @@ "x64" ], "dev": true, - "libc": [ - "musl" - ], "license": "MPL-2.0", "optional": true, "os": [ diff --git a/package.json b/package.json index 440576d..8fcd8bb 100644 --- a/package.json +++ b/package.json @@ -24,6 +24,7 @@ "author": "", "license": "ISC", "dependencies": { + "istanbul-lib-coverage": "^3.2.2", "openapi-mock-validator": "^0.3.0", "puppeteer": "^25.3.0" }, diff --git a/src/mergeCoverage.js b/src/mergeCoverage.js new file mode 100644 index 0000000..f7172e7 --- /dev/null +++ b/src/mergeCoverage.js @@ -0,0 +1,26 @@ +import libCoverage from 'istanbul-lib-coverage'; + +/** + * Combines per-shard Istanbul coverage into one map. + * + * This is the one artifact that needed no bespoke merge logic: summing hit + * counts across runs of the same code is exactly what CoverageMap.merge does, + * and it is the same operation `nyc merge` performs. + * + * Nulls are skipped rather than rejected — a shard that collected no coverage + * (a filtered run, or one that never loaded instrumented code) reads back as + * null and simply does not contribute. + * + * Inputs are cloned before merging. `istanbul-lib-coverage`'s `FileCoverage` + * wraps a plain coverage object by reference instead of copying it, so the + * first shard's raw object would otherwise become the map's live storage and + * get mutated in place (`this.data.s = ...`) once a later shard's counts for + * the same file are merged in. + */ +export function mergeCoverage(coverageObjects) { + const map = libCoverage.createCoverageMap({}); + for (const coverage of coverageObjects) { + if (coverage) map.merge(structuredClone(coverage)); + } + return map.toJSON(); +} diff --git a/tests/mergeCoverage.test.js b/tests/mergeCoverage.test.js new file mode 100644 index 0000000..97bfdb7 --- /dev/null +++ b/tests/mergeCoverage.test.js @@ -0,0 +1,70 @@ +import { describe, it, expect } from 'vitest'; +import { mergeCoverage } from '../src/mergeCoverage.js'; + +// Minimal but structurally valid Istanbul file coverage. istanbul-lib-coverage +// validates the shape, so the maps cannot be omitted. +function fileCoverage(path, statementHits, fnHits = 0) { + return { + path, + statementMap: { 0: { start: { line: 1, column: 0 }, end: { line: 1, column: 10 } } }, + fnMap: { + 0: { + name: 'f', + decl: { start: { line: 1, column: 0 }, end: { line: 1, column: 1 } }, + loc: { start: { line: 1, column: 0 }, end: { line: 3, column: 1 } }, + }, + }, + branchMap: {}, + s: { 0: statementHits }, + f: { 0: fnHits }, + b: {}, + }; +} + +describe('mergeCoverage', () => { + it('sums statement hits for the same file across shards', () => { + const merged = mergeCoverage([ + { '/app/src/a.js': fileCoverage('/app/src/a.js', 1) }, + { '/app/src/a.js': fileCoverage('/app/src/a.js', 2) }, + ]); + expect(merged['/app/src/a.js'].s[0]).toBe(3); + }); + + it('sums function hits for the same file across shards', () => { + const merged = mergeCoverage([ + { '/app/src/a.js': fileCoverage('/app/src/a.js', 1, 4) }, + { '/app/src/a.js': fileCoverage('/app/src/a.js', 1, 5) }, + ]); + expect(merged['/app/src/a.js'].f[0]).toBe(9); + }); + + it('unions files that only one shard touched', () => { + const merged = mergeCoverage([ + { '/app/src/a.js': fileCoverage('/app/src/a.js', 1) }, + { '/app/src/b.js': fileCoverage('/app/src/b.js', 7) }, + ]); + expect(Object.keys(merged).sort()).toEqual(['/app/src/a.js', '/app/src/b.js']); + expect(merged['/app/src/b.js'].s[0]).toBe(7); + }); + + // A shard with no coverage file reads back as null and must not break the merge. + it('skips null and undefined entries', () => { + const merged = mergeCoverage([ + null, + { '/app/src/a.js': fileCoverage('/app/src/a.js', 2) }, + undefined, + ]); + expect(merged['/app/src/a.js'].s[0]).toBe(2); + }); + + it('returns an empty map for no input', () => { + expect(mergeCoverage([])).toEqual({}); + expect(mergeCoverage([null])).toEqual({}); + }); + + it('does not mutate its inputs', () => { + const first = { '/app/src/a.js': fileCoverage('/app/src/a.js', 1) }; + mergeCoverage([first, { '/app/src/a.js': fileCoverage('/app/src/a.js', 5) }]); + expect(first['/app/src/a.js'].s[0]).toBe(1); + }); +}); From 2ac151e695901e82962a38f3f851e41623986ad5 Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Wed, 19 Aug 2026 22:16:48 +0200 Subject: [PATCH 12/26] feat(merge): associative report merge with consistency validation Adds src/mergeReports.js: mergeRunReports combines shard reports while validating only cross-shard consistency (schemaVersion, fingerprint, shard total, duplicate index/test id). Completeness is intentionally left to the separately-exported findMissingShards, called later by the merge command, so a partial merge stays associative. Also exports reportTimings (wall vs compute time) and reportTotals (executed/notRun vs discovered count), both derived rather than stored. --- src/mergeReports.js | 130 +++++++++++++++++++++++++ tests/mergeReports.test.js | 189 +++++++++++++++++++++++++++++++++++++ 2 files changed, 319 insertions(+) create mode 100644 src/mergeReports.js create mode 100644 tests/mergeReports.test.js diff --git a/src/mergeReports.js b/src/mergeReports.js new file mode 100644 index 0000000..0e8331a --- /dev/null +++ b/src/mergeReports.js @@ -0,0 +1,130 @@ +/** + * Combines shard reports into one report of the same shape. + * + * Only *consistency* is validated here — the things that stay true under a + * partial merge. Completeness (is 1..total all present?) is checked by + * findMissingShards, called from the merge command, because a 2-of-3 merge is a + * legal intermediate value: rejecting it here would make + * mergeRunReports([mergeRunReports([a, b]), c]) throw and destroy + * associativity, which is the property that proves no test is lost or doubled. + */ +export function mergeRunReports(reports) { + if (!reports.length) { + throw new Error( + 'No shard reports to merge. Expected /*/run.json or /run.json.' + ); + } + + const versions = [...new Set(reports.map((r) => r.schemaVersion))]; + if (versions.length > 1) { + throw new Error( + `Shard reports disagree on schemaVersion (${versions.sort().join(', ')}). ` + + 'Every shard job must run the same twd-cli version.' + ); + } + + const fingerprints = new Set(reports.map((r) => r.discovery.fingerprint)); + if (fingerprints.size > 1) { + throw new Error( + 'Shard reports discovered different test sets, so they cannot be merged. ' + + 'Either tests are registered conditionally (a feature flag, a date, ' + + 'Math.random), or the shard jobs did not build the same code.' + ); + } + + const shards = reports.flatMap((r) => r.shards); + + const totals = [...new Set(shards.map((s) => s.total))]; + if (totals.length > 1) { + throw new Error( + `Shard reports disagree on shard total (${totals.sort((a, b) => a - b).join(', ')}). ` + + 'Every shard job must pass the same --shard total.' + ); + } + + const byIndex = new Set(); + for (const shard of shards) { + if (byIndex.has(shard.index)) { + throw new Error(`Shard ${shard.index}/${shard.total} appears more than once.`); + } + byIndex.add(shard.index); + } + + const tests = []; + const testIds = new Set(); + for (const report of reports) { + for (const test of report.tests) { + if (testIds.has(test.id)) { + throw new Error( + `Test id "${test.id}" appears in more than one shard — the shard slices overlap.` + ); + } + testIds.add(test.id); + tests.push(test); + } + } + + const first = reports[0]; + + return { + schemaVersion: first.schemaVersion, + // Copy before sorting: sort mutates, and the input reports are the caller's. + shards: [...shards].sort((a, b) => a.index - b.index), + discovery: first.discovery, + selection: first.selection, + // Identical across shards, guaranteed by the fingerprint check above. + handlers: first.handlers, + tests, + contracts: { + configured: first.contracts.configured, + partial: reports.some((r) => r.contracts.partial), + results: reports.flatMap((r) => r.contracts.results), + skipped: reports.flatMap((r) => r.contracts.skipped), + }, + }; +} + +/** + * Shard indices in 1..total that no report accounted for. + * + * A gap almost always means a shard job died before uploading its artifact. It + * must be loud: a silent 3-of-4 merge reads as a complete green run. + */ +export function findMissingShards(report) { + const total = report.shards[0]?.total ?? 0; + const present = new Set(report.shards.map((s) => s.index)); + const missing = []; + for (let i = 1; i <= total; i++) { + if (!present.has(i)) missing.push(i); + } + return missing; +} + +/** + * Wall clock (what the developer waited) and compute (what was paid for). + * + * Derived rather than stored, so the two can never drift out of agreement with + * the per-shard timestamps they come from. + */ +export function reportTimings(report) { + const starts = report.shards.map((s) => Date.parse(s.startedAt)); + const ends = report.shards.map((s) => Date.parse(s.endedAt)); + return { + wallMs: Math.max(...ends) - Math.min(...starts), + computeMs: report.shards.reduce((sum, s) => sum + s.durationMs, 0), + }; +} + +/** + * Executed and not-run totals, plus whether they account for every discovered + * test. An inconsistent result points at a shard-math bug, not user error. + */ +export function reportTotals(report) { + const executed = report.shards.reduce((sum, s) => sum + s.executed, 0); + const notRun = report.shards.reduce((sum, s) => sum + s.notRun, 0); + return { + executed, + notRun, + consistent: executed + notRun === report.discovery.totalTests, + }; +} diff --git a/tests/mergeReports.test.js b/tests/mergeReports.test.js new file mode 100644 index 0000000..9d05b09 --- /dev/null +++ b/tests/mergeReports.test.js @@ -0,0 +1,189 @@ +import { describe, it, expect } from 'vitest'; +import { + mergeRunReports, + findMissingShards, + reportTimings, + reportTotals, +} from '../src/mergeReports.js'; + +const HANDLERS = [ + { id: 's1', name: 'Login', parent: null, type: 'suite' }, + { id: 't1', name: 'a', parent: 's1', type: 'test' }, + { id: 't2', name: 'b', parent: 's1', type: 'test' }, + { id: 't3', name: 'c', parent: 's1', type: 'test' }, +]; + +const FINGERPRINT = 'sha256:deadbeef'; + +function makeReport(index, overrides = {}) { + const { + total = 3, + tests = [{ id: `t${index}`, status: 'pass' }], + startedAt = `2026-08-19T10:00:0${index}.000Z`, + endedAt = `2026-08-19T10:00:1${index}.000Z`, + durationMs = 10_000, + executed = 1, + notRun = 0, + failed = 0, + stoppedEarly = false, + coverageFile = 'coverage.json', + contracts = { configured: true, partial: false, results: [], skipped: [] }, + fingerprint = FINGERPRINT, + schemaVersion = 1, + totalTests = 3, + } = overrides; + + return { + schemaVersion, + shards: [{ + index, total, startedAt, endedAt, durationMs, + executed, notRun, failed, stoppedEarly, coverageFile, recording: null, + }], + discovery: { totalTests, fingerprint }, + selection: { filters: [] }, + handlers: HANDLERS, + tests, + contracts, + }; +} + +describe('mergeRunReports', () => { + it('concatenates tests across shards', () => { + const merged = mergeRunReports([makeReport(1), makeReport(2), makeReport(3)]); + expect(merged.tests.map((t) => t.id)).toEqual(['t1', 't2', 't3']); + }); + + it('sorts shard descriptors by index regardless of input order', () => { + const merged = mergeRunReports([makeReport(3), makeReport(1), makeReport(2)]); + expect(merged.shards.map((s) => s.index)).toEqual([1, 2, 3]); + }); + + it('keeps the single-report shape', () => { + const merged = mergeRunReports([makeReport(1), makeReport(2), makeReport(3)]); + expect(Object.keys(merged).sort()).toEqual( + ['contracts', 'discovery', 'handlers', 'schemaVersion', 'selection', 'shards', 'tests'], + ); + expect(merged.handlers).toEqual(HANDLERS); + expect(merged.discovery).toEqual({ totalTests: 3, fingerprint: FINGERPRINT }); + }); + + // The property that proves nothing is lost or doubled. It only holds because + // completeness is checked outside this function. + it('is associative', () => { + const a = makeReport(1); + const b = makeReport(2); + const c = makeReport(3); + expect(mergeRunReports([mergeRunReports([a, b]), c])) + .toEqual(mergeRunReports([a, b, c])); + expect(mergeRunReports([a, mergeRunReports([b, c])])) + .toEqual(mergeRunReports([a, b, c])); + }); + + it('accepts a partial merge without complaining about gaps', () => { + const merged = mergeRunReports([makeReport(1), makeReport(2)]); + expect(merged.shards.map((s) => s.index)).toEqual([1, 2]); + }); + + it('concatenates contract results and skipped entries', () => { + const merged = mergeRunReports([ + makeReport(1, { contracts: { configured: true, partial: false, results: [{ alias: 'a' }], skipped: [{ alias: 'x' }] } }), + makeReport(2, { contracts: { configured: true, partial: false, results: [{ alias: 'b' }], skipped: [] } }), + ]); + expect(merged.contracts.results).toEqual([{ alias: 'a' }, { alias: 'b' }]); + expect(merged.contracts.skipped).toEqual([{ alias: 'x' }]); + }); + + it('ORs the contracts partial flag', () => { + const partial = makeReport(2, { contracts: { configured: true, partial: true, results: [], skipped: [] } }); + expect(mergeRunReports([makeReport(1), partial]).contracts.partial).toBe(true); + expect(mergeRunReports([makeReport(1), makeReport(2)]).contracts.partial).toBe(false); + }); + + it('throws on an empty input', () => { + expect(() => mergeRunReports([])).toThrow(/No shard reports/); + }); + + it('throws when schema versions disagree', () => { + expect(() => mergeRunReports([makeReport(1), makeReport(2, { schemaVersion: 2 })])) + .toThrow(/schemaVersion/); + }); + + // The safety net: shards that saw different test sets must never be combined. + it('throws when fingerprints disagree', () => { + expect(() => mergeRunReports([makeReport(1), makeReport(2, { fingerprint: 'sha256:other' })])) + .toThrow(/different test sets/); + }); + + it('throws when shard totals disagree', () => { + expect(() => mergeRunReports([makeReport(1), makeReport(2, { total: 4 })])) + .toThrow(/shard total/); + }); + + it('throws when the same shard index appears twice', () => { + expect(() => mergeRunReports([makeReport(1), makeReport(1)])) + .toThrow(/more than once/); + }); + + it('throws when a test id appears in two shards', () => { + expect(() => mergeRunReports([ + makeReport(1, { tests: [{ id: 'dup', status: 'pass' }] }), + makeReport(2, { tests: [{ id: 'dup', status: 'pass' }] }), + ])).toThrow(/"dup" appears in more than one shard/); + }); + + it('does not mutate the input reports', () => { + const a = makeReport(1); + const b = makeReport(2); + mergeRunReports([b, a]); + expect(a.shards).toHaveLength(1); + expect(b.shards[0].index).toBe(2); + }); +}); + +describe('findMissingShards', () => { + it('returns an empty array when every shard is present', () => { + expect(findMissingShards(mergeRunReports([makeReport(1), makeReport(2), makeReport(3)]))) + .toEqual([]); + }); + + it('names the gaps', () => { + expect(findMissingShards(mergeRunReports([makeReport(1), makeReport(3)]))).toEqual([2]); + expect(findMissingShards(mergeRunReports([makeReport(2)]))).toEqual([1, 3]); + }); +}); + +describe('reportTimings', () => { + // Wall clock is what the developer waited; compute is what was paid for. + it('reports wall clock as the span and compute as the sum', () => { + const merged = mergeRunReports([makeReport(1), makeReport(2), makeReport(3)]); + // starts 10:00:01..03, ends 10:00:11..13 -> span 12s; 3 x 10s compute + expect(reportTimings(merged)).toEqual({ wallMs: 12_000, computeMs: 30_000 }); + }); + + it('makes wall and compute equal for a single shard', () => { + const single = mergeRunReports([makeReport(1, { total: 1 })]); + const { wallMs, computeMs } = reportTimings(single); + expect(wallMs).toBe(computeMs); + }); +}); + +describe('reportTotals', () => { + it('sums executed and notRun and confirms they account for discovery', () => { + const merged = mergeRunReports([makeReport(1), makeReport(2), makeReport(3)]); + expect(reportTotals(merged)).toEqual({ executed: 3, notRun: 0, consistent: true }); + }); + + it('flags totals that do not add up to the discovered count', () => { + const merged = mergeRunReports([makeReport(1), makeReport(2)]); + expect(reportTotals(merged).consistent).toBe(false); + }); + + it('counts a bailed shard\'s notRun', () => { + const merged = mergeRunReports([ + makeReport(1), + makeReport(2, { executed: 1, notRun: 1, stoppedEarly: true, failed: 1 }), + makeReport(3), + ]); + expect(reportTotals(merged)).toEqual({ executed: 3, notRun: 1, consistent: false }); + }); +}); From 96d9dacc6576ce36e94970728e08dff0445a8baf Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Wed, 19 Aug 2026 22:24:22 +0200 Subject: [PATCH 13/26] fix(merge): pin first-wins semantics with a test, correct handlers comment MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review found the existing tests couldn't distinguish taking discovery/ selection/handlers/contracts.configured from the first shard report vs the last, since every fixture shared identical values for those fields. Adds a test that makes them differ (while keeping fingerprint/schemaVersion/ total consistent, since a partial merge is legal) and asserts first-wins. Also corrects a comment claiming the fingerprint check guarantees handler identity across shards — fingerprintTests only hashes the ordered test-id list and filters, not handler metadata, so that guarantee doesn't exist. --- src/mergeReports.js | 5 ++++- tests/mergeReports.test.js | 19 +++++++++++++++++++ 2 files changed, 23 insertions(+), 1 deletion(-) diff --git a/src/mergeReports.js b/src/mergeReports.js index 0e8331a..0fcee5f 100644 --- a/src/mergeReports.js +++ b/src/mergeReports.js @@ -72,7 +72,10 @@ export function mergeRunReports(reports) { shards: [...shards].sort((a, b) => a.index - b.index), discovery: first.discovery, selection: first.selection, - // Identical across shards, guaranteed by the fingerprint check above. + // Invariant across shards in practice: every shard enumerates the same app. + // Not proven by the fingerprint, which covers the ordered test-id list and + // the filters, not handler metadata. Taking the first is the documented + // contract, pinned by a test. handlers: first.handlers, tests, contracts: { diff --git a/tests/mergeReports.test.js b/tests/mergeReports.test.js index 9d05b09..8233967 100644 --- a/tests/mergeReports.test.js +++ b/tests/mergeReports.test.js @@ -67,6 +67,25 @@ describe('mergeRunReports', () => { expect(merged.discovery).toEqual({ totalTests: 3, fingerprint: FINGERPRINT }); }); + // Locks the documented first-wins semantics. These fields are invariant across + // valid shards by construction, so nothing upstream distinguishes first from + // last — only this test does. + it('takes discovery, selection, handlers and contracts.configured from the first report', () => { + const first = makeReport(1, { totalTests: 3 }); + const second = makeReport(2, { totalTests: 3 }); + second.discovery = { ...second.discovery, totalTests: 99 }; + second.selection = { filters: ['not-the-first'] }; + second.handlers = [{ id: 'other', name: 'Other', parent: null, type: 'suite' }]; + second.contracts = { ...second.contracts, configured: false }; + + const merged = mergeRunReports([first, second]); + + expect(merged.discovery.totalTests).toBe(3); + expect(merged.selection).toEqual({ filters: [] }); + expect(merged.handlers).toEqual(HANDLERS); + expect(merged.contracts.configured).toBe(true); + }); + // The property that proves nothing is lost or doubled. It only holds because // completeness is checked outside this function. it('is associative', () => { From 8f0ef787e0404416c8c3d109ae0c69ebea1318f6 Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Wed, 19 Aug 2026 23:20:00 +0200 Subject: [PATCH 14/26] feat(summary): add per-shard breakdown and wall vs compute duration --- src/testSummary.js | 16 +++++++++++++- tests/testSummary.test.js | 44 +++++++++++++++++++++++++++++++++++++++ 2 files changed, 59 insertions(+), 1 deletion(-) diff --git a/src/testSummary.js b/src/testSummary.js index 7497005..c7d717c 100644 --- a/src/testSummary.js +++ b/src/testSummary.js @@ -7,6 +7,8 @@ export function formatRunComplete({ notRun = 0, stoppedEarly = false, maxFailures, + shards = null, + computeMs = null, }) { const passed = testStatus.filter((t) => t.status === 'pass').length; const failed = testStatus.filter((t) => t.status === 'fail').length; @@ -18,7 +20,19 @@ export function formatRunComplete({ ` Passed: ${passed} | Failed: ${failed} | Skipped: ${skipped}`, ]; if (notRun > 0) lines.push(` Not run: ${notRun}`); - lines.push(` Duration: ${duration}s`); + + // A merged run has two meaningful durations: the span the developer waited, + // and the compute it consumed. A single run has only one, and its line must + // stay byte-identical to what it has always printed. + const merged = Array.isArray(shards) && shards.length > 1; + if (merged) { + const compute = (computeMs / 1000).toFixed(1); + lines.push(` Duration: ${duration}s wall | ${compute}s across ${shards.length} shards`); + const cells = shards.map((s) => `${s.index} ${s.failed > 0 ? '✗' : '✓'}${s.executed}`); + lines.push(` Shards: ${cells.join(' | ')}`); + } else { + lines.push(` Duration: ${duration}s`); + } const failures = testStatus.filter((t) => t.status === 'fail'); if (failures.length > 0) { diff --git a/tests/testSummary.test.js b/tests/testSummary.test.js index 4950d75..98a17a1 100644 --- a/tests/testSummary.test.js +++ b/tests/testSummary.test.js @@ -178,3 +178,47 @@ describe('formatRunComplete', () => { expect(block).not.toContain('0 test(s) were not run'); }); }); + +describe('formatRunComplete with shards', () => { + const handlers = [ + { id: 's1', name: 'Login', parent: null, type: 'suite' }, + { id: 't1', name: 'works', parent: 's1', type: 'test' }, + ]; + const testStatus = [{ id: 't1', status: 'pass' }]; + + function shard(index, overrides = {}) { + return { index, total: 4, executed: 30, failed: 0, notRun: 0, stoppedEarly: false, ...overrides }; + } + + it('adds a shard breakdown line when more than one shard merged', () => { + const output = formatRunComplete({ + testStatus, handlers, durationMs: 38_200, computeMs: 134_200, + shards: [shard(1), shard(2, { failed: 3 }), shard(3), shard(4)], + }); + expect(output).toContain('Shards: 1 ✓30 | 2 ✗30 | 3 ✓30 | 4 ✓30'); + }); + + it('reports wall clock and compute separately for a merged run', () => { + const output = formatRunComplete({ + testStatus, handlers, durationMs: 38_200, computeMs: 134_200, + shards: [shard(1), shard(2)], + }); + expect(output).toContain('Duration: 38.2s wall | 134.2s across 2 shards'); + }); + + // The existing single-run format must not shift. + it('keeps the plain duration line when there are no shards', () => { + const output = formatRunComplete({ testStatus, handlers, durationMs: 4_200 }); + expect(output).toContain('Duration: 4.2s'); + expect(output).not.toContain('wall'); + expect(output).not.toContain('Shards:'); + }); + + it('keeps the plain duration line for a single shard', () => { + const output = formatRunComplete({ + testStatus, handlers, durationMs: 4_200, computeMs: 4_200, shards: [shard(1, { total: 1 })], + }); + expect(output).toContain('Duration: 4.2s'); + expect(output).not.toContain('Shards:'); + }); +}); From 182ea0153b39b74f0083efa10710b6fe5a368a0d Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Wed, 19 Aug 2026 23:31:09 +0200 Subject: [PATCH 15/26] feat(run): slice tests by shard and write a run report artifact --shard takes a round-robin slice of the ordered test ids after --test filters resolve, then writes .twd/run/run.json (plus coverage.json when collected) for a later merge. Two gates are relaxed for sharded runs only, and both reduce to the original expression when shard is absent: - coverage: !hasFailures becomes (sharded || !hasFailures), because hasFailures is per shard and would let three green shards contribute coverage while a red fourth contributes none. - contracts: !stoppedEarly becomes (sharded || !stoppedEarly), with the result flagged partial so merge can say what is missing. A sharded run deliberately writes neither .nyc_output/out.json nor the contract markdown report: one shard's fraction sitting at either path would masquerade as the whole run's. Co-Authored-By: Claude Opus 5 (1M context) --- src/index.js | 146 +++++++++++++++++----- tests/runTests.test.js | 272 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 390 insertions(+), 28 deletions(-) diff --git a/src/index.js b/src/index.js index 3d4f348..837735d 100644 --- a/src/index.js +++ b/src/index.js @@ -11,6 +11,9 @@ import { selectTestIds } from './filterTests.js'; import { explainError } from './diagnostics.js'; import { orderedTestIds, chunk } from './testOrder.js'; import { resolveRecordFilename } from './recordFilename.js'; +import { selectShardIds } from './shard.js'; +import { buildRunReport } from './runReport.js'; +import { writeRunReport, DEFAULT_REPORT_DIR, COVERAGE_FILE } from './reportFiles.js'; import { assertFfmpegAvailable, applyRecordingFraming, @@ -43,7 +46,8 @@ function recordedFileSize(absPath) { } export async function runTests(options = {}) { - const { testFilters = [], recordOverrides = {} } = options; + const { testFilters = [], recordOverrides = {}, shard = null, reportDir = null } = options; + const sharded = Boolean(shard); let browser; let config; let startedAt = null; @@ -52,6 +56,7 @@ export async function runTests(options = {}) { let recorder = null; let recordOutput = null; let recordOutputPath = null; + let recordingInfo = null; // Stops the screencast at most once. Must always run before browser.close(): // if the browser goes first, ffmpeg is orphaned and the file is truncated. @@ -161,7 +166,22 @@ export async function runTests(options = {}) { } // Resolve the ordered id list to run: the filter result, or all tests. - const baseIds = selectedIds ?? orderedTestIds(registeredHandlers); + // + // allTestIds is the full ordered list, before filtering or slicing. This is + // what the fingerprint hashes and what discovery.totalTests reports, so + // every shard agrees on it regardless of which slice it took. + const allTestIds = orderedTestIds(registeredHandlers); + const filteredIds = selectedIds ?? allTestIds; + const baseIds = sharded + ? selectShardIds(filteredIds, shard.index, shard.total) + : filteredIds; + + if (sharded) { + console.log( + `Shard ${shard.index}/${shard.total}: running ${baseIds.length} of ${filteredIds.length} test(s).` + ); + } + const chunks = chunk(baseIds, config.chunkSize); // Recording starts here, not earlier: the output path is fixed up front by @@ -274,12 +294,14 @@ export async function runTests(options = {}) { 'Chrome only emits video frames when the page repaints, so a run with no visible changes (or no tests) records nothing.' ); } else { + recordingInfo = { file: recordOutput, bytes: recordedFileSize(recordOutputPath) }; console.log(`Recorded ${executed} test(s) to ${recordOutput}`); } } const testStatus = partialStatus; - const durationMs = Date.now() - startedAt; + const endedAt = Date.now(); + const durationMs = endedAt - startedAt; const notRun = baseIds.length - executed; // Exit with appropriate code @@ -292,8 +314,19 @@ export async function runTests(options = {}) { } } - // Contract validation (skipped on an early stop — the data is partial) - if (!stoppedEarly && config.contracts && config.contracts.length > 0) { + // Contract validation. A sharded run validates even after an early stop and + // flags the result partial, so merge can say exactly what is missing rather + // than silently dropping a shard's worth of mocks. A non-sharded run keeps + // skipping, exactly as before. + const contractsConfigured = Boolean(config.contracts && config.contracts.length > 0); + let contractsBlock = { + configured: contractsConfigured, + partial: false, + results: [], + skipped: [], + }; + + if (contractsConfigured && (sharded || !stoppedEarly)) { if (collectedMocks.size === 0) { console.log('\nNo mocks collected — ensure twd-js supports contract collection'); } @@ -303,46 +336,78 @@ export async function runTests(options = {}) { hasFailures = true; } - // Write markdown report for CI/PR integration - if (config.contractReportPath) { + contractsBlock = { + configured: true, + partial: stoppedEarly, + results: validationOutput.results, + skipped: validationOutput.skipped, + }; + + if (stoppedEarly) { + console.log('\n⚠ Contract data is partial — this shard stopped early.'); + } + + // Write markdown report for CI/PR integration. + // + // Only a whole run produces a meaningful markdown report. Under sharding + // each shard would overwrite the others with a quarter of the picture, so + // `merge` writes it instead. + if (config.contractReportPath && !sharded) { const reportPath = path.resolve(workingDir, config.contractReportPath); - const reportDir = path.dirname(reportPath); - if (!fs.existsSync(reportDir)) { - fs.mkdirSync(reportDir, { recursive: true }); + const reportDirPath = path.dirname(reportPath); + if (!fs.existsSync(reportDirPath)) { + fs.mkdirSync(reportDirPath, { recursive: true }); } const markdown = generateContractMarkdown(validationOutput); fs.writeFileSync(reportPath, markdown); console.log(`Contract report written to ${config.contractReportPath}`); } - } else if (stoppedEarly && config.contracts && config.contracts.length > 0) { + } else if (contractsConfigured && stoppedEarly) { console.log('\nSkipping contract validation — run stopped early (partial data).'); } - // Handle code coverage if enabled (skipped when a --test filter is active) + // Handle code coverage if enabled. + // + // The filter gate is unchanged: a --test filter still suppresses coverage, + // because a filtered run's number is a misleading project-wide figure. A + // shard slice is not a filter. + // + // The failure gate is relaxed for sharded runs only. hasFailures is per + // shard, so applying it here would let three green shards write coverage + // while a red fourth writes none — a merged report that looks complete but + // is missing a quarter of the code paths. `merge` applies the gate to the + // true global result instead. if (selectedIds && config.coverage) { console.log('Skipping coverage collection (test filter active).'); } - if (config.coverage && !hasFailures && !selectedIds) { - const coverage = await page.evaluate(() => window.__coverage__); - if (coverage) { - const coverageDir = path.resolve(workingDir, config.coverageDir); - const nycDir = path.resolve(workingDir, config.nycOutputDir); - - if (!fs.existsSync(nycDir)) { - fs.mkdirSync(nycDir, { recursive: true }); - } - if (!fs.existsSync(coverageDir)) { - fs.mkdirSync(coverageDir, { recursive: true }); - } - const coveragePath = path.join(nycDir, 'out.json'); - fs.writeFileSync(coveragePath, JSON.stringify(coverage)); - console.log(`Code coverage data written to ${coveragePath}`); - } else { + let coverageData = null; + if (config.coverage && !selectedIds && (sharded || !hasFailures)) { + coverageData = await page.evaluate(() => window.__coverage__); + if (!coverageData) { console.log('No code coverage data found.'); } } + // A sharded run's coverage goes to the report dir and nowhere else. Writing + // it to .nyc_output/out.json — the path nyc reads by default — would let one + // shard's partial data masquerade as the whole run's. + if (coverageData && !sharded) { + const coverageDir = path.resolve(workingDir, config.coverageDir); + const nycDir = path.resolve(workingDir, config.nycOutputDir); + + if (!fs.existsSync(nycDir)) { + fs.mkdirSync(nycDir, { recursive: true }); + } + if (!fs.existsSync(coverageDir)) { + fs.mkdirSync(coverageDir, { recursive: true }); + } + + const coveragePath = path.join(nycDir, 'out.json'); + fs.writeFileSync(coveragePath, JSON.stringify(coverageData)); + console.log(`Code coverage data written to ${coveragePath}`); + } + await browser.close(); // The run-complete block is always the last output of a completed run @@ -356,6 +421,31 @@ export async function runTests(options = {}) { maxFailures: config.maxFailures, })); + // Written last, and only for a sharded run. A run that threw never gets + // here on purpose: its artifact stays absent, and `merge` reports the gap as + // "a shard job likely failed before uploading", which is the accurate + // diagnosis. A half-written report would be a worse lie. + if (sharded) { + const dir = reportDir ?? DEFAULT_REPORT_DIR; + const report = buildRunReport({ + shard, + startedAt, + endedAt, + allTestIds, + filters: testFilters, + handlers, + tests: testStatus, + executed, + notRun, + stoppedEarly, + coverageFile: coverageData ? COVERAGE_FILE : null, + recording: recordingInfo, + contracts: contractsBlock, + }); + const { reportPath } = writeRunReport(dir, report, coverageData); + console.log(`Shard report written to ${reportPath}`); + } + return hasFailures; } catch (error) { diff --git a/tests/runTests.test.js b/tests/runTests.test.js index fbf5da3..a774316 100644 --- a/tests/runTests.test.js +++ b/tests/runTests.test.js @@ -1183,3 +1183,275 @@ describe("runTests pacing", () => { }); }); + +describe('runTests sharding', () => { + function fourTests() { + return { + handlers: [ + { id: '1', name: 'a', type: 'test' }, + { id: '2', name: 'b', type: 'test' }, + { id: '3', name: 'c', type: 'test' }, + { id: '4', name: 'd', type: 'test' }, + ], + testStatus: [{ id: '2', status: 'pass' }, { id: '4', status: 'pass' }], + }; + } + + function runJson() { + const call = vi.mocked(fs.writeFileSync).mock.calls + .find(([file]) => String(file).endsWith('run.json')); + return call ? JSON.parse(call[1]) : null; + } + + // runJson() and the not.toHaveBeenCalled() assertions read fs call history, so + // this block cannot inherit another describe's. loadConfig has to be re-stubbed + // too: clearAllMocks keeps implementations, so without this the previous block's + // recording-enabled config (with a pace) leaks in. + beforeEach(() => { + vi.clearAllMocks(); + vi.mocked(loadConfig).mockReturnValue({ ...defaultMockConfig }); + vi.mocked(assertFfmpegAvailable).mockReset(); + vi.mocked(fs.statSync).mockReset(); + vi.spyOn(console, 'log').mockImplementation(() => {}); + }); + + afterEach(() => { + vi.restoreAllMocks(); + }); + + // Round-robin: shard 2 of 2 takes the odd indices, i.e. the 2nd and 4th ids. + it('runs only its round-robin slice', async () => { + const { handlers, testStatus } = fourTests(); + const page = createMockPage({ handlers, testStatus }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + + await runTests({ shard: { index: 2, total: 2 } }); + + expect(page.evaluate).toHaveBeenCalledWith(expect.any(Function), 2, ['2', '4']); + }); + + it('writes a run report to the default report dir', async () => { + const { handlers, testStatus } = fourTests(); + const page = createMockPage({ handlers, testStatus }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + + await runTests({ shard: { index: 2, total: 2 } }); + + expect(fs.mkdirSync).toHaveBeenCalledWith('./.twd/run', { recursive: true }); + const report = runJson(); + expect(report.schemaVersion).toBe(1); + expect(report.shards[0]).toMatchObject({ index: 2, total: 2, executed: 2 }); + expect(report.discovery.totalTests).toBe(4); + expect(report.tests.map((t) => t.id)).toEqual(['2', '4']); + }); + + it('honors --report-dir', async () => { + const { handlers, testStatus } = fourTests(); + const page = createMockPage({ handlers, testStatus }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + + await runTests({ shard: { index: 1, total: 1 }, reportDir: './out' }); + + expect(fs.mkdirSync).toHaveBeenCalledWith('./out', { recursive: true }); + }); + + it('writes no report when not sharded', async () => { + const { handlers, testStatus } = fourTests(); + const page = createMockPage({ handlers, testStatus }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + + await runTests(); + + expect(runJson()).toBeNull(); + }); + + // 3 tests across 4 shards leaves the fourth with nothing to run. It must + // still write a valid report, or merge sees a gap it cannot explain. + it('writes a valid empty report when its slice is empty', async () => { + const handlers = [ + { id: '1', name: 'a', type: 'test' }, + { id: '2', name: 'b', type: 'test' }, + { id: '3', name: 'c', type: 'test' }, + ]; + const page = createMockPage({ handlers, testStatus: [] }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + + const hasFailures = await runTests({ shard: { index: 4, total: 4 } }); + + expect(hasFailures).toBe(false); + const report = runJson(); + expect(report.tests).toEqual([]); + expect(report.shards[0]).toMatchObject({ index: 4, total: 4, executed: 0, failed: 0 }); + // The fingerprint still covers the whole suite, so an empty shard merges + // cleanly with the three that ran something. + expect(report.discovery.totalTests).toBe(3); + }); + + // Filters resolve first, then the filtered list is sharded. A filtered run's + // coverage is a misleading project-wide number, sharded or not. + it('skips coverage when a filter is combined with a shard', async () => { + const { handlers, testStatus } = fourTests(); + const page = createMockPage({ handlers, testStatus }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + vi.mocked(loadConfig).mockReturnValue({ ...defaultMockConfig, coverage: true }); + + await runTests({ shard: { index: 1, total: 2 }, testFilters: ['a'] }); + + const files = vi.mocked(fs.writeFileSync).mock.calls.map(([f]) => String(f)); + expect(files.some((f) => f.endsWith('coverage.json'))).toBe(false); + expect(runJson().shards[0].coverageFile).toBeNull(); + expect(runJson().selection.filters).toEqual(['a']); + }); +}); + +describe('runTests non-regression: non-sharded behavior is unchanged', () => { + beforeEach(() => { + vi.clearAllMocks(); + vi.mocked(loadConfig).mockReturnValue({ ...defaultMockConfig }); + vi.mocked(assertFfmpegAvailable).mockReset(); + vi.mocked(fs.statSync).mockReset(); + vi.spyOn(console, 'log').mockImplementation(() => {}); + }); + + afterEach(() => { + vi.restoreAllMocks(); + }); + + // The !hasFailures coverage gate is relaxed only for sharded runs. A plain + // failing run must still write nothing, exactly as in 1.4.0. + it('writes no coverage when a non-sharded run fails', async () => { + const handlers = [{ id: '1', name: 'a', type: 'test' }]; + const page = createMockPage({ handlers, testStatus: [{ id: '1', status: 'fail', error: 'boom' }] }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + vi.mocked(loadConfig).mockReturnValue({ ...defaultMockConfig, coverage: true }); + + await runTests(); + + expect(fs.writeFileSync).not.toHaveBeenCalled(); + }); + + // The successful coverage write. Splitting its if/else apart is the most + // delicate edit in the sharding change and nothing else exercised this block, + // so pin the destination: nyc reads .nyc_output/out.json by default. + it('still writes coverage to .nyc_output/out.json when a non-sharded run passes', async () => { + const handlers = [{ id: '1', name: 'a', type: 'test' }]; + const page = createMockPage({ handlers, testStatus: [{ id: '1', status: 'pass' }] }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + vi.mocked(loadConfig).mockReturnValue({ ...defaultMockConfig, coverage: true }); + + await runTests(); + + const files = vi.mocked(fs.writeFileSync).mock.calls.map(([f]) => String(f)); + expect(files).toHaveLength(1); + expect(files[0].includes('.nyc_output')).toBe(true); + expect(files[0].endsWith('out.json')).toBe(true); + }); + + // Likewise the early-stop contract skip. + it('still skips contract validation when a non-sharded run stops early', async () => { + const handlers = [{ id: '1', name: 'a', type: 'test' }]; + const page = createMockPage({ handlers, testStatus: [{ id: '1', status: 'fail', error: 'boom' }] }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + vi.mocked(loadConfig).mockReturnValue({ + ...defaultMockConfig, maxFailures: 1, contracts: [{ source: 'api.json' }], + }); + vi.mocked(loadContracts).mockResolvedValue([]); + + await runTests(); + + expect(validateMocks).not.toHaveBeenCalled(); + }); + + // The markdown report gained a `&& !sharded` gate. A plain run must still + // write it, and this is the only test that executes that block at all. + it('still writes the contract markdown report on a non-sharded run', async () => { + const handlers = [{ id: '1', name: 'a', type: 'test' }]; + const page = createMockPage({ handlers, testStatus: [{ id: '1', status: 'pass' }] }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + vi.mocked(loadConfig).mockReturnValue({ + ...defaultMockConfig, + contracts: [{ source: 'api.json' }], + contractReportPath: './contract-report.md', + }); + vi.mocked(loadContracts).mockResolvedValue([]); + vi.mocked(validateMocks).mockReturnValue({ results: [], skipped: [] }); + vi.mocked(printContractReport).mockReturnValue(false); + + await runTests(); + + const files = vi.mocked(fs.writeFileSync).mock.calls.map(([f]) => String(f)); + expect(files.some((f) => f.endsWith('contract-report.md'))).toBe(true); + }); +}); + +describe('runTests sharded behavior changes', () => { + beforeEach(() => { + vi.clearAllMocks(); + vi.mocked(loadConfig).mockReturnValue({ ...defaultMockConfig }); + vi.mocked(assertFfmpegAvailable).mockReset(); + vi.mocked(fs.statSync).mockReset(); + vi.spyOn(console, 'log').mockImplementation(() => {}); + }); + + afterEach(() => { + vi.restoreAllMocks(); + }); + + it('writes coverage for a sharded run that fails', async () => { + const handlers = [{ id: '1', name: 'a', type: 'test' }]; + const page = createMockPage({ handlers, testStatus: [{ id: '1', status: 'fail', error: 'boom' }] }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + vi.mocked(loadConfig).mockReturnValue({ ...defaultMockConfig, coverage: true }); + + await runTests({ shard: { index: 1, total: 1 } }); + + const files = vi.mocked(fs.writeFileSync).mock.calls.map(([f]) => String(f)); + expect(files.some((f) => f.endsWith('coverage.json'))).toBe(true); + // Never the path nyc reads by default: a shard's partial coverage there + // would masquerade as the whole run's. + expect(files.some((f) => f.includes('.nyc_output'))).toBe(false); + }); + + // Every shard would overwrite the others with a fraction of the picture, so + // the markdown report is merge's job. The run.json still gets written. + it('writes no contract markdown report when sharded', async () => { + const handlers = [{ id: '1', name: 'a', type: 'test' }]; + const page = createMockPage({ handlers, testStatus: [{ id: '1', status: 'pass' }] }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + vi.mocked(loadConfig).mockReturnValue({ + ...defaultMockConfig, + contracts: [{ source: 'api.json' }], + contractReportPath: './contract-report.md', + }); + vi.mocked(loadContracts).mockResolvedValue([]); + vi.mocked(validateMocks).mockReturnValue({ results: [], skipped: [] }); + vi.mocked(printContractReport).mockReturnValue(false); + + await runTests({ shard: { index: 1, total: 1 } }); + + const files = vi.mocked(fs.writeFileSync).mock.calls.map(([f]) => String(f)); + expect(files.some((f) => f.endsWith('contract-report.md'))).toBe(false); + expect(files.some((f) => f.endsWith('run.json'))).toBe(true); + }); + + it('validates contracts on a sharded early stop and marks them partial', async () => { + const handlers = [{ id: '1', name: 'a', type: 'test' }]; + const page = createMockPage({ handlers, testStatus: [{ id: '1', status: 'fail', error: 'boom' }] }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + vi.mocked(loadConfig).mockReturnValue({ + ...defaultMockConfig, maxFailures: 1, contracts: [{ source: 'api.json' }], + }); + vi.mocked(loadContracts).mockResolvedValue([]); + vi.mocked(validateMocks).mockReturnValue({ results: [{ alias: 'a' }], skipped: [] }); + vi.mocked(printContractReport).mockReturnValue(false); + + await runTests({ shard: { index: 1, total: 1 } }); + + expect(validateMocks).toHaveBeenCalled(); + const call = vi.mocked(fs.writeFileSync).mock.calls + .find(([file]) => String(file).endsWith('run.json')); + const report = JSON.parse(call[1]); + expect(report.contracts).toMatchObject({ configured: true, partial: true }); + expect(report.contracts.results).toEqual([{ alias: 'a' }]); + }); +}); From 84f68fe394ed8f435349bb67e9870fa347e63d86 Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Wed, 19 Aug 2026 23:45:36 +0200 Subject: [PATCH 16/26] feat(cli): add the merge command and wire shard flags through bin --- bin/twd-cli.js | 35 ++++- src/mergeCommand.js | 142 +++++++++++++++++++ tests/mergeCommand.test.js | 270 +++++++++++++++++++++++++++++++++++++ 3 files changed, 444 insertions(+), 3 deletions(-) create mode 100644 src/mergeCommand.js create mode 100644 tests/mergeCommand.test.js diff --git a/bin/twd-cli.js b/bin/twd-cli.js index a5b0d84..fb08ffb 100755 --- a/bin/twd-cli.js +++ b/bin/twd-cli.js @@ -1,14 +1,31 @@ #!/usr/bin/env node import { runTests } from '../src/index.js'; -import { parseRunArgs } from '../src/parseArgs.js'; +import { parseRunArgs, parseMergeArgs } from '../src/parseArgs.js'; +import { runMerge } from '../src/mergeCommand.js'; const command = process.argv[2]; if (command === 'run') { try { - const { testFilters, record } = parseRunArgs(process.argv.slice(3)); - const hasFailures = await runTests({ testFilters, recordOverrides: record }); + const { testFilters, record, shard, reportDir } = parseRunArgs(process.argv.slice(3)); + const hasFailures = await runTests({ + testFilters, + recordOverrides: record, + shard, + reportDir, + }); + process.exit(hasFailures ? 1 : 0); + } catch (error) { + if (!error?.reported) { + console.error(error?.message ?? String(error)); + } + process.exit(1); + } +} else if (command === 'merge') { + try { + const { dir, out } = parseMergeArgs(process.argv.slice(3)); + const hasFailures = runMerge({ dir, out }); process.exit(hasFailures ? 1 : 0); } catch (error) { if (!error?.reported) { @@ -26,13 +43,25 @@ Usage: contains (case-insensitive). Repeatable; multiple --test values are OR'd. npx twd-cli run --record Record the run to a video file + npx twd-cli run --shard 2/4 Run only this shard's slice of the suite + and write a report to ./.twd/run + npx twd-cli merge Merge shard reports from into one + report, and exit 1 if the whole run failed Examples: npx twd-cli run --test "shows error" npx twd-cli run --test "Login" --test "Signup" + npx twd-cli run --shard 2/4 + npx twd-cli merge .twd/shards Options: --test "" Filter tests by "suite > test" path (repeatable, OR'd) + --shard / Run slice i of n. Each shard discovers the whole + suite and takes every nth test, so the test count + never has to be known in advance. Implies a report. + --report-dir Where to write the shard report (default ./.twd/run) + --out merge only: where to write the merged report + (default ./.twd/merged-run.json) --record Record the run to a video file (requires ffmpeg) --record-dir Output directory (default ./twd-artifacts) --record-speed Playback speed, e.g. 0.5 for half speed diff --git a/src/mergeCommand.js b/src/mergeCommand.js new file mode 100644 index 0000000..be4e3a9 --- /dev/null +++ b/src/mergeCommand.js @@ -0,0 +1,142 @@ +import fs from 'fs'; +import path from 'path'; +import { loadConfig } from './config.js'; +import { + readShardReports, + readShardCoverage, + DEFAULT_MERGED_OUT, +} from './reportFiles.js'; +import { + mergeRunReports, + findMissingShards, + reportTimings, + reportTotals, +} from './mergeReports.js'; +import { mergeCoverage } from './mergeCoverage.js'; +import { formatRunComplete } from './testSummary.js'; +import { printContractReport } from './contractReport.js'; +import { generateContractMarkdown } from './contractMarkdown.js'; + +/** + * Joins per-shard reports into one and reports on the whole run. + * + * This function owns the run's exit code. Shard jobs each exit 1 on their own + * failures, so the workflow only reaches here with `if: !cancelled()`, and the + * merged verdict is the one that counts. + */ +export function runMerge({ dir, out = null } = {}) { + if (!dir) { + throw new Error('Usage: twd-cli merge [--out ]'); + } + + const config = loadConfig(); + const workingDir = process.cwd(); + + const found = readShardReports(dir); + if (found.length === 0) { + throw new Error( + `No shard reports found in ${dir}. ` + + 'Expected /*/run.json (the actions/download-artifact layout) or /run.json.' + ); + } + + // readShardReports returns readdir order, so twd-run-10 sorts before + // twd-run-2. mergeRunReports sorts shards[] by index but concatenates tests + // in argument order, so without this the merged report's tests and its shard + // list disagree about ordering. + found.sort((a, b) => (a.report.shards[0]?.index ?? 0) - (b.report.shards[0]?.index ?? 0)); + + const merged = mergeRunReports(found.map((f) => f.report)); + + // Completeness is enforced here rather than inside mergeRunReports, which must + // stay associative. A gap is never a warning: a silent 3-of-4 merge reads as a + // complete green run. + const missing = findMissingShards(merged); + if (missing.length > 0) { + const total = merged.shards[0].total; + throw new Error( + `Missing shard report(s): ${missing.map((i) => `${i}/${total}`).join(', ')}. ` + + 'A shard job likely failed before uploading its artifact — check that the ' + + 'upload step runs with `if: always()`.' + ); + } + + const outPath = path.resolve(workingDir, out ?? DEFAULT_MERGED_OUT); + fs.mkdirSync(path.dirname(outPath), { recursive: true }); + fs.writeFileSync(outPath, `${JSON.stringify(merged, null, 2)}\n`); + console.log(`Merged report written to ${outPath}`); + + let hasFailures = merged.tests.some((test) => test.status === 'fail'); + + if (merged.contracts.configured) { + const validationOutput = { + results: merged.contracts.results, + skipped: merged.contracts.skipped, + }; + if (printContractReport(validationOutput)) { + hasFailures = true; + } + if (merged.contracts.partial) { + console.warn( + 'Warning: contract data is partial — at least one shard stopped early, so ' + + 'some mocks were never collected.' + ); + } + if (config.contractReportPath) { + const reportPath = path.resolve(workingDir, config.contractReportPath); + fs.mkdirSync(path.dirname(reportPath), { recursive: true }); + fs.writeFileSync(reportPath, generateContractMarkdown(validationOutput)); + console.log(`Contract report written to ${config.contractReportPath}`); + } + } + + // A red run yields no coverage — the same policy a single run has always had, + // but keyed on the whole merged result instead of one shard's. + if (config.coverage) { + const coverages = found.map((f) => + readShardCoverage(f.dir, f.report.shards[0]?.coverageFile ?? null) + ); + const contributors = coverages.filter(Boolean).length; + + if (contributors === 0) { + console.log('No coverage data found in any shard.'); + } else if (hasFailures) { + console.log( + `Skipping merged coverage — the run has failures ` + + `(${contributors}/${found.length} shard(s) had data).` + ); + } else { + const nycDir = path.resolve(workingDir, config.nycOutputDir); + fs.mkdirSync(nycDir, { recursive: true }); + fs.writeFileSync(path.join(nycDir, 'out.json'), JSON.stringify(mergeCoverage(coverages))); + console.log( + `Coverage merged from ${contributors}/${found.length} shards to ` + + `${config.nycOutputDir}/out.json` + ); + } + } + + const totals = reportTotals(merged); + if (!totals.consistent) { + console.warn( + `Warning: shard totals do not add up — ${totals.executed} executed + ` + + `${totals.notRun} not run != ${merged.discovery.totalTests} discovered. ` + + 'This points at a shard-slicing bug, not at your tests.' + ); + } + + const timings = reportTimings(merged); + console.log(''); + console.log(formatRunComplete({ + testStatus: merged.tests, + handlers: merged.handlers, + durationMs: timings.wallMs, + computeMs: timings.computeMs, + notRun: totals.notRun, + shards: merged.shards, + stoppedEarly: merged.shards.some((s) => s.stoppedEarly), + maxFailures: config.maxFailures, + })); + + return hasFailures; +} diff --git a/tests/mergeCommand.test.js b/tests/mergeCommand.test.js new file mode 100644 index 0000000..2eb020f --- /dev/null +++ b/tests/mergeCommand.test.js @@ -0,0 +1,270 @@ +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; + +vi.mock('fs'); +vi.mock('../src/config.js', () => ({ loadConfig: vi.fn() })); +vi.mock('../src/contractReport.js', () => ({ printContractReport: vi.fn() })); +vi.mock('../src/reportFiles.js', async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, readShardReports: vi.fn(), readShardCoverage: vi.fn() }; +}); + +import fs from 'fs'; +import { loadConfig } from '../src/config.js'; +import { printContractReport } from '../src/contractReport.js'; +import { readShardReports, readShardCoverage } from '../src/reportFiles.js'; +import { runMerge } from '../src/mergeCommand.js'; + +const HANDLERS = [ + { id: 's1', name: 'Login', parent: null, type: 'suite' }, + { id: 't1', name: 'a', parent: 's1', type: 'test' }, + { id: 't2', name: 'b', parent: 's1', type: 'test' }, +]; + +function shardReport(index, overrides = {}) { + const { total = 2, tests = [{ id: `t${index}`, status: 'pass' }], failed = 0 } = overrides; + return { + schemaVersion: 1, + shards: [{ + index, total, + startedAt: `2026-08-19T10:00:0${index}.000Z`, + endedAt: `2026-08-19T10:00:1${index}.000Z`, + durationMs: 10_000, + executed: 1, notRun: 0, failed, + stoppedEarly: false, coverageFile: 'coverage.json', recording: null, + }], + discovery: { totalTests: 2, fingerprint: 'sha256:same' }, + selection: { filters: [] }, + handlers: HANDLERS, + tests, + contracts: { configured: false, partial: false, results: [], skipped: [] }, + }; +} + +const baseConfig = { + coverage: true, + nycOutputDir: './.nyc_output', + maxFailures: 10, +}; + +function writtenFiles() { + return vi.mocked(fs.writeFileSync).mock.calls.map(([f]) => String(f)); +} + +describe('runMerge', () => { + beforeEach(() => { + vi.clearAllMocks(); + vi.mocked(loadConfig).mockReturnValue({ ...baseConfig }); + vi.mocked(readShardCoverage).mockReturnValue(null); + vi.spyOn(console, 'log').mockImplementation(() => {}); + vi.spyOn(console, 'warn').mockImplementation(() => {}); + }); + + afterEach(() => { + vi.restoreAllMocks(); + }); + + it('requires a directory', () => { + expect(() => runMerge({})).toThrow(/Usage: twd-cli merge/); + }); + + it('errors when no shard reports were found', () => { + vi.mocked(readShardReports).mockReturnValue([]); + expect(() => runMerge({ dir: '.twd/shards' })).toThrow(/No shard reports found/); + }); + + // The failure that must never be silent: three green shards and one that + // never uploaded would otherwise read as a complete green run. + it('errors and names the gap when a shard is missing', () => { + vi.mocked(readShardReports).mockReturnValue([{ dir: 'a', report: shardReport(1) }]); + expect(() => runMerge({ dir: '.twd/shards' })).toThrow(/Missing shard report\(s\): 2\/2/); + }); + + it('mentions if: always() in the missing-shard message', () => { + vi.mocked(readShardReports).mockReturnValue([{ dir: 'a', report: shardReport(2) }]); + expect(() => runMerge({ dir: '.twd/shards' })).toThrow(/if: always\(\)/); + }); + + it('writes the merged report to the default path', () => { + vi.mocked(readShardReports).mockReturnValue([ + { dir: 'a', report: shardReport(1) }, + { dir: 'b', report: shardReport(2) }, + ]); + + expect(runMerge({ dir: '.twd/shards' })).toBe(false); + + const merged = writtenFiles().find((f) => f.endsWith('merged-run.json')); + expect(merged).toBeDefined(); + }); + + it('honors --out', () => { + vi.mocked(readShardReports).mockReturnValue([ + { dir: 'a', report: shardReport(1) }, + { dir: 'b', report: shardReport(2) }, + ]); + + runMerge({ dir: '.twd/shards', out: 'custom.json' }); + + expect(writtenFiles().some((f) => f.endsWith('custom.json'))).toBe(true); + }); + + // readShardReports returns readdir order, which is lexicographic, so + // twd-run-10 comes back before twd-run-2. mergeRunReports sorts shards[] by + // index but concatenates tests in argument order, so without a sort here the + // merged artifact's test order and its shard list would disagree. + it('merges shards in index order whatever order they were found in', () => { + vi.mocked(readShardReports).mockReturnValue([ + { dir: 'twd-run-2', report: shardReport(2) }, + { dir: 'twd-run-1', report: shardReport(1) }, + ]); + + runMerge({ dir: '.twd/shards' }); + + const call = vi.mocked(fs.writeFileSync).mock.calls + .find(([f]) => String(f).endsWith('merged-run.json')); + const merged = JSON.parse(String(call[1])); + expect(merged.tests.map((t) => t.id)).toEqual(['t1', 't2']); + expect(merged.shards.map((s) => s.index)).toEqual([1, 2]); + }); + + it('returns true when any shard had a failing test', () => { + vi.mocked(readShardReports).mockReturnValue([ + { dir: 'a', report: shardReport(1) }, + { dir: 'b', report: shardReport(2, { tests: [{ id: 't2', status: 'fail', error: 'boom' }], failed: 1 }) }, + ]); + + expect(runMerge({ dir: '.twd/shards' })).toBe(true); + }); + + it('merges coverage when the run is green', () => { + vi.mocked(readShardReports).mockReturnValue([ + { dir: 'a', report: shardReport(1) }, + { dir: 'b', report: shardReport(2) }, + ]); + vi.mocked(readShardCoverage).mockReturnValue({}); + + runMerge({ dir: '.twd/shards' }); + + expect(writtenFiles().some((f) => f.includes('.nyc_output'))).toBe(true); + }); + + // The user's rule, applied to the true global result rather than one shard's. + it('skips merged coverage when the run is red', () => { + vi.mocked(readShardReports).mockReturnValue([ + { dir: 'a', report: shardReport(1) }, + { dir: 'b', report: shardReport(2, { tests: [{ id: 't2', status: 'fail', error: 'boom' }], failed: 1 }) }, + ]); + vi.mocked(readShardCoverage).mockReturnValue({}); + + runMerge({ dir: '.twd/shards' }); + + expect(writtenFiles().some((f) => f.includes('.nyc_output'))).toBe(false); + }); + + it('reports how many shards contributed coverage', () => { + const log = vi.spyOn(console, 'log'); + vi.mocked(readShardReports).mockReturnValue([ + { dir: 'a', report: shardReport(1) }, + { dir: 'b', report: shardReport(2) }, + ]); + vi.mocked(readShardCoverage).mockReturnValueOnce({}).mockReturnValueOnce(null); + + runMerge({ dir: '.twd/shards' }); + + expect(log.mock.calls.flat().join('\n')).toMatch(/Coverage merged from 1\/2 shards/); + }); + + it('says so when no shard had coverage', () => { + const log = vi.spyOn(console, 'log'); + vi.mocked(readShardReports).mockReturnValue([ + { dir: 'a', report: shardReport(1) }, + { dir: 'b', report: shardReport(2) }, + ]); + + runMerge({ dir: '.twd/shards' }); + + expect(log.mock.calls.flat().join('\n')).toMatch(/No coverage data found/); + }); + + it('returns true when contracts report an error-mode violation', () => { + const withContracts = (i) => ({ + ...shardReport(i), + contracts: { configured: true, partial: false, results: [{ alias: 'a' }], skipped: [] }, + }); + vi.mocked(readShardReports).mockReturnValue([ + { dir: 'a', report: withContracts(1) }, + { dir: 'b', report: withContracts(2) }, + ]); + vi.mocked(printContractReport).mockReturnValue(true); + + expect(runMerge({ dir: '.twd/shards' })).toBe(true); + }); + + it('warns when contract data is partial', () => { + const warn = vi.spyOn(console, 'warn'); + const partial = (i, isPartial) => ({ + ...shardReport(i), + contracts: { configured: true, partial: isPartial, results: [], skipped: [] }, + }); + vi.mocked(readShardReports).mockReturnValue([ + { dir: 'a', report: partial(1, false) }, + { dir: 'b', report: partial(2, true) }, + ]); + vi.mocked(printContractReport).mockReturnValue(false); + + runMerge({ dir: '.twd/shards' }); + + expect(warn.mock.calls.flat().join('\n')).toMatch(/contract data is partial/i); + }); + + // A sharded run skips the markdown report on purpose — each shard would + // overwrite the others with a quarter of the picture. Merge is where the whole + // picture exists, so this is the only place it can be written. + it('writes the contract markdown report that sharded runs skip', () => { + vi.mocked(loadConfig).mockReturnValue({ ...baseConfig, contractReportPath: './contract-report.md' }); + const configured = (i) => ({ + ...shardReport(i), + contracts: { configured: true, partial: false, results: [], skipped: [] }, + }); + vi.mocked(readShardReports).mockReturnValue([ + { dir: 'a', report: configured(1) }, + { dir: 'b', report: configured(2) }, + ]); + vi.mocked(printContractReport).mockReturnValue(false); + + runMerge({ dir: '.twd/shards' }); + + expect(writtenFiles().some((f) => f.endsWith('contract-report.md'))).toBe(true); + }); + + // Executed + not-run has to account for every discovered test. When it does + // not, the shard math dropped tests on the floor and nothing else would say so. + it('warns when the shard totals do not account for every discovered test', () => { + const warn = vi.spyOn(console, 'warn'); + const wrongTotal = (i) => ({ + ...shardReport(i), + discovery: { totalTests: 3, fingerprint: 'sha256:same' }, + }); + vi.mocked(readShardReports).mockReturnValue([ + { dir: 'a', report: wrongTotal(1) }, + { dir: 'b', report: wrongTotal(2) }, + ]); + + runMerge({ dir: '.twd/shards' }); + + expect(warn.mock.calls.flat().join('\n')).toMatch(/shard totals do not add up/); + }); + + it('prints the merged run-complete block with a shard breakdown', () => { + const log = vi.spyOn(console, 'log'); + vi.mocked(readShardReports).mockReturnValue([ + { dir: 'a', report: shardReport(1) }, + { dir: 'b', report: shardReport(2) }, + ]); + + runMerge({ dir: '.twd/shards' }); + + const output = log.mock.calls.flat().join('\n'); + expect(output).toContain('--- Run complete ---'); + expect(output).toContain('Shards: 1 ✓1 | 2 ✓1'); + }); +}); From e80fbfd1d5aa4276e60a5706d3138bfc96d8e3df Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Wed, 19 Aug 2026 23:54:54 +0200 Subject: [PATCH 17/26] ci: verify sharded runs and merge end to end --- .github/workflows/e2e.yml | 113 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 113 insertions(+) diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml index e155947..ca8b88b 100644 --- a/.github/workflows/e2e.yml +++ b/.github/workflows/e2e.yml @@ -93,3 +93,116 @@ jobs: echo "" >> "$REPORT_PATH" echo "[View full report →](${RUN_URL})" >> "$REPORT_PATH" gh pr comment "${PR_NUMBER}" --body-file "$REPORT_PATH" + + e2e-sharded: + runs-on: ubuntu-latest + + strategy: + # Without this, the first red shard cancels its siblings and the merge job + # sees gaps it cannot distinguish from a crashed shard. + fail-fast: false + matrix: + shard: [1, 2] + + steps: + - name: Checkout repo + uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd # v5 + + - name: Setup Node.js + uses: actions/setup-node@a0853c24544627f65ddf259abe73b1d18a591444 # v5 + with: + node-version: 24 + cache: npm + + - name: Install CLI dependencies + run: npm ci + + - name: Install test-example-app dependencies + working-directory: test-example-app + run: npm install + + - name: Install mock service worker + working-directory: test-example-app + run: npx twd-js init public --save + + - name: Cache Puppeteer browsers + uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4 + with: + path: ~/.cache/puppeteer + key: ${{ runner.os }}-puppeteer-${{ hashFiles('package-lock.json') }} + restore-keys: | + ${{ runner.os }}-puppeteer- + + - name: Install Chrome for Puppeteer + run: npx puppeteer browsers install chrome + + - name: Start dev server + working-directory: test-example-app + run: | + nohup npx vite --host > vite.log 2>&1 & + npx wait-on http://localhost:5173 --timeout 30000 + + - name: Run TWD tests for this shard + working-directory: test-example-app + run: node ../bin/twd-cli.js run --shard ${{ matrix.shard }}/2 + + - name: Upload shard report + # Always: a red shard must still upload, or merge cannot tell "this shard + # failed" from "this shard never ran". + if: always() + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 + with: + name: twd-run-${{ matrix.shard }} + path: test-example-app/.twd/run + if-no-files-found: error + + e2e-merge: + runs-on: ubuntu-latest + needs: [e2e-sharded] + # Runs even though a shard job may have exited 1. Without this the merged + # summary — the point of the exercise — is never printed. + if: ${{ !cancelled() }} + + steps: + - name: Checkout repo + uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd # v5 + + - name: Setup Node.js + uses: actions/setup-node@a0853c24544627f65ddf259abe73b1d18a591444 # v5 + with: + node-version: 24 + cache: npm + + - name: Install CLI dependencies + run: npm ci + + - name: Download shard reports + uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4 + with: + pattern: twd-run-* + path: test-example-app/.twd/shards + + - name: Merge shard reports + working-directory: test-example-app + run: node ../bin/twd-cli.js merge .twd/shards + + - name: Verify the merged report + working-directory: test-example-app + run: | + if [ ! -f .twd/merged-run.json ]; then + echo "ERROR: merged report not generated" + exit 1 + fi + node -e " + const r = require('./.twd/merged-run.json'); + if (r.shards.length !== 2) { + console.error('ERROR: expected 2 shards, got ' + r.shards.length); + process.exit(1); + } + if (r.tests.length !== r.discovery.totalTests) { + console.error('ERROR: ' + r.tests.length + ' merged tests but ' + + r.discovery.totalTests + ' discovered'); + process.exit(1); + } + console.log('Merged ' + r.tests.length + ' tests from ' + r.shards.length + ' shards'); + " From e0304d66a650a3dd656a846547adbcd5f98c129b Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Thu, 20 Aug 2026 00:05:19 +0200 Subject: [PATCH 18/26] ci: fix unfalsifiable assertions in the merge verify step Drop the shards.length !== 2 check (already enforced by findMissingShards before merged-run.json is even written, so it can never fire) and the file-existence check's implied claim of being a gate (the merge step has no if: always(), so a nonzero exit never reaches here). Replace with an executed-vs-merged-tests check that can actually catch a shard-slicing bug that silently drops tests. --- .github/workflows/e2e.yml | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/.github/workflows/e2e.yml b/.github/workflows/e2e.yml index ca8b88b..c94ce2b 100644 --- a/.github/workflows/e2e.yml +++ b/.github/workflows/e2e.yml @@ -102,6 +102,8 @@ jobs: # sees gaps it cannot distinguish from a crashed shard. fail-fast: false matrix: + # The "2" here must match the shard total below (--shard N/2). Bump both + # together, or shards will disagree on the total and merge will reject them. shard: [1, 2] steps: @@ -189,20 +191,26 @@ jobs: - name: Verify the merged report working-directory: test-example-app run: | + # Cheap sanity guard, not a real gate: the "Merge shard reports" step + # above has no `if: always()`, so we only reach here after `twd-cli + # merge` exited 0 — and runMerge only writes merged-run.json once it + # has decided to succeed (src/mergeCommand.js). This check protects + # against that assumption changing later, but it cannot fail today. if [ ! -f .twd/merged-run.json ]; then echo "ERROR: merged report not generated" exit 1 fi node -e " const r = require('./.twd/merged-run.json'); - if (r.shards.length !== 2) { - console.error('ERROR: expected 2 shards, got ' + r.shards.length); - process.exit(1); - } if (r.tests.length !== r.discovery.totalTests) { console.error('ERROR: ' + r.tests.length + ' merged tests but ' + r.discovery.totalTests + ' discovered'); process.exit(1); } + const executed = r.shards.reduce((sum, s) => sum + s.executed, 0); + if (executed !== r.tests.length) { + console.error('ERROR: shards report ' + executed + ' executed but merged ' + r.tests.length + ' tests'); + process.exit(1); + } console.log('Merged ' + r.tests.length + ' tests from ' + r.shards.length + ' shards'); " From 29e5d1e0e6f5298593c365f6c27d40f95560f322 Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Thu, 20 Aug 2026 09:18:39 +0200 Subject: [PATCH 19/26] chore(release): 1.5.0-beta.0 --- CHANGELOG.md | 16 +++++++++ README.md | 85 +++++++++++++++++++++++++++++++++++++++++++++++ package-lock.json | 4 +-- package.json | 8 ++++- 4 files changed, 110 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 3fe713b..7b67d61 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,19 @@ +## 1.5.0-beta.0 (2026-08-19) + +* feat(shard): `--shard /` runs one slice of the suite so a run can be split across parallel CI jobs. Each shard discovers the whole suite itself and takes every nth test, so the test count never has to be known in advance +* feat(shard): a sharded run writes `run.json` and `coverage.json` to `./.twd/run` (`--report-dir` to change it) — the first machine-readable output twd-cli has had +* feat(merge): `npx twd-cli merge ` joins shard reports into one report covering test results, coverage and contract validation, prints a single summary with a per-shard breakdown, and owns the exit code +* feat(merge): a missing shard report is an error naming the gap, not a silently incomplete report. Shards also fingerprint the test list they discovered, so shards that saw different test sets refuse to merge +* note: no behavior change without `--shard`. A plain run writes the same files, prints the same output, and exits the same way as 1.4.0 + +Sharding needs three things right in the workflow: `fail-fast: false` on the +matrix, `if: always()` on the shard's artifact upload, and +`if: ${{ !cancelled() }}` on the merge job. See "Sharding across CI jobs" in the +README. + +This is a prerelease, published under the `beta` dist-tag: +`npm install twd-cli@beta`. + ## 1.4.0 (2026-07-28) * feat(record): video recording for twd-cli runs (#13) ([94c21e0](https://github.com/BRIKEV/twd-cli/commit/94c21e0)), closes [#13](https://github.com/BRIKEV/twd-cli/issues/13) diff --git a/README.md b/README.md index 580037e..f40b6d8 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,7 @@ CI/CD runner for [TWD (Test while developing)](https://brikev.github.io/twd/) - [Recording](#recording): capture a run to video, paced so it is watchable - [Contract Validation](#contract-validation): check your mocks against OpenAPI specs - [CI/CD Integration](#cicd-integration): GitHub Action and custom setups +- [Sharding across CI jobs](#sharding-across-ci-jobs): split a run across parallel CI jobs - [How It Works](#how-it-works) - [Requirements](#requirements) @@ -267,6 +268,90 @@ jobs: run: npm run collect:coverage:text ``` +## Sharding across CI jobs + +A single run walks the whole suite in one browser. `--shard` splits it across +parallel CI jobs instead. + +```bash +npx twd-cli run --shard 2/4 # "I am job 2 of 4" +npx twd-cli merge .twd/shards # join the reports back together +``` + +The `4` is how many jobs you are running, **not** how many tests exist. You never +need to know the test count: each shard boots its own browser, discovers the whole +suite exactly as a normal run does, and keeps every 4th test. Add tests and the +same 4 jobs just split more of them. + +Each shard writes `run.json` and `coverage.json` to `./.twd/run` (change it with +`--report-dir`). `merge` reads the downloaded shard directories, combines the test +results, coverage and contract validation, prints one summary, and exits non-zero +if anything failed anywhere. + +```yaml +jobs: + test: + strategy: + fail-fast: false # or one red shard cancels the rest + matrix: + shard: [1, 2, 3, 4] + steps: + # ...checkout, npm ci, chrome, dev server... + - run: npx twd-cli run --shard ${{ matrix.shard }}/4 + - uses: actions/upload-artifact@v4 + if: always() # a red shard must still upload + with: + name: twd-run-${{ matrix.shard }} + path: .twd/run + + merge: + needs: [test] + if: ${{ !cancelled() }} # runs even though a shard went red + steps: + - uses: actions/checkout@v5 + - uses: actions/setup-node@v5 + with: + node-version: 24 + cache: npm + - run: npm ci + - uses: actions/download-artifact@v4 + with: + pattern: twd-run-* + path: .twd/shards + - run: npx twd-cli merge .twd/shards +``` + +Those three conditions are easy to miss and each one breaks the run: +`fail-fast: false` stops a red shard cancelling its siblings, `if: always()` on +upload keeps a red shard's report, and `if: ${{ !cancelled() }}` on merge lets the +summary print at all. + +### Notes + +- **Coverage.** Each shard writes its own `coverage.json`; `merge` combines them + into `.nyc_output/out.json` — but only when the whole run is green, matching how + a single run behaves. `merge` reports how many shards contributed. +- **Missing shards are an error.** If a shard job dies before uploading, `merge` + refuses and names the gap rather than silently reporting 3 of 4 shards as a + complete green run. +- **Tests must register identically in every job.** Each shard fingerprints the + test list it discovered and `merge` verifies they match. Registering tests + conditionally — behind a feature flag, a date, `Math.random()` — makes the + fingerprints diverge and `merge` will say so. +- **`maxFailures` is per shard.** Four shards at the default of 10 can reach 40 + failures between them before all four bail. +- **`--test` and `--shard` compose:** filters resolve first, then the filtered list + is sharded. As with any filtered run, coverage is skipped. +- **Recording** produces one clip per shard; they are not concatenated. +- **A missing shard leaves no merged report on disk.** `merge` throws before it + writes `.twd/merged-run.json`, so a CI step that uploads that path with + `if: always()` will find nothing when a shard is missing. The error message on + stderr is the diagnosis in that case. +- **`record.filename` collides under sharding.** Only the *derived* recording + filename is per-shard. If `record.filename` is set explicitly in + `twd.config.json`, every shard writes to the same video path. Use the derived + name, or a per-shard `--record-dir`, when recording a sharded run. + ## Contract Validation Validate your test mocks against OpenAPI specs to catch drift between your mocks and the real API. When a mock response doesn't match the spec, you'll see errors like: diff --git a/package-lock.json b/package-lock.json index 908160f..bc71b51 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "twd-cli", - "version": "1.4.0", + "version": "1.5.0-beta.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "twd-cli", - "version": "1.4.0", + "version": "1.5.0-beta.0", "license": "ISC", "dependencies": { "istanbul-lib-coverage": "^3.2.2", diff --git a/package.json b/package.json index 8fcd8bb..986998d 100644 --- a/package.json +++ b/package.json @@ -1,12 +1,18 @@ { "name": "twd-cli", - "version": "1.4.0", + "version": "1.5.0-beta.0", "description": "CLI tool for running TWD tests with Puppeteer", "type": "module", "main": "src/index.js", "bin": { "twd-cli": "./bin/twd-cli.js" }, + "files": [ + "bin", + "src", + "README.md", + "LICENSE" + ], "scripts": { "test": "vitest", "test:ci": "vitest --run --coverage", From 58a609630855cb8f5731e91de60e40b98a9f092c Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Thu, 20 Aug 2026 10:01:27 +0200 Subject: [PATCH 20/26] fix(shard): key cross-shard identity on test paths and positions, not ids MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit twd-js mints test ids with Math.random() at registration time (twd/src/runner.ts:52), so an id is a per-page-load nonce. Every shard boots its own browser, so no two shards ever agree on an id for the same test. The design built identity on those ids, which broke the feature three ways and made a fourth check a no-op: - discovery.fingerprint hashed the ordered id list, so it could never match across shards and `merge` refused every correct multi-shard run — blaming the user's app for conditional registration. - A merged report keeps only the first shard's handlers, so buildTestPath could not resolve anything from shards 2..n: every failed or retried test from them printed as a raw random id in the merged summary, the one output the merge exists to produce. - mergeRunReports' overlap check was keyed on ids that by construction never collide, so it looked like a guard while proving nothing. Identity is now two fields with distinct jobs. tests[].path is the "suite > test" string, resolved inside the shard that ran the test (the only place its handler map is valid); it is what the fingerprint hashes and what the summary displays. tests[].index is the test's position in the discovered order — deterministic across shards, and the identity key, because a path cannot serve: duplicate test names share one and may legally land in different shards. Fingerprinting paths is also strictly stronger than ids: a conditionally registered test still drops out of the ordered list. Two more findings from the same review: - discovery.totalTests was the unfiltered count while executed/notRun counted the filtered-and-sliced list, so `--test` with `--shard` printed a bogus "shard totals do not add up ... points at a shard-slicing bug". The report now records selection.selectedTests — what the shards actually divide — and reportTotals compares against that. - The "No mocks collected — ensure twd-js supports contract collection" hint fired for every shard whose slice exercised no mocks, advertising a version problem that does not exist on the happy path of a sharded CI run. Gated on !sharded. mergeRunReports also now validates schemaVersion against REPORT_SCHEMA_VERSION rather than only checking that shards agree: reports from a newer twd-cli agree with each other and were being mis-merged silently. REPORT_SCHEMA_VERSION 1 -> 2. The fingerprint's meaning changed, so a v1 and a v2 report would both claim to be mergeable and surface as "different test sets" instead of "run the same twd-cli version". Nothing is published yet. Packaging: add CHANGELOG.md to the files allowlist and document the allowlist in the 1.5.0-beta.0 entry (tarball 209 kB -> 33 kB, 99 files -> 25). No behavior change without --shard: a non-sharded run writes no report, so path/index never reach it, and formatRunComplete falls back to the handler lookup for entries that carry no path. Verified end to end against test-example-app with two real browsers: shards that share zero handler ids now produce identical fingerprints, `merge` succeeds and prints real "suite > test" paths for a shard-2 failure, and the pre-fix binary still reproduces the refusal on the same app. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 1 + README.md | 8 +- ...26-08-19-shardable-run-artifacts-design.md | 21 +++- package.json | 1 + src/index.js | 15 ++- src/mergeCommand.js | 2 +- src/mergeReports.js | 50 +++++++-- src/runReport.js | 50 +++++++-- src/testSummary.js | 19 +++- tests/mergeCommand.test.js | 93 ++++++++++++++-- tests/mergeReports.test.js | 73 ++++++++++--- tests/runReport.test.js | 67 ++++++++++-- tests/runTests.test.js | 102 +++++++++++++++++- tests/testSummary.test.js | 35 ++++++ 14 files changed, 473 insertions(+), 64 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 7b67d61..cac57a1 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,7 @@ * feat(shard): a sharded run writes `run.json` and `coverage.json` to `./.twd/run` (`--report-dir` to change it) — the first machine-readable output twd-cli has had * feat(merge): `npx twd-cli merge ` joins shard reports into one report covering test results, coverage and contract validation, prints a single summary with a per-shard breakdown, and owns the exit code * feat(merge): a missing shard report is an error naming the gap, not a silently incomplete report. Shards also fingerprint the test list they discovered, so shards that saw different test sets refuse to merge +* chore(packaging): a `files` allowlist in package.json — the published package is now just `bin/`, `src/`, `README.md`, `CHANGELOG.md` and `LICENSE`. `tests/`, `test-example-app/`, `docs/` and the repo tooling were all being published and no longer are, taking the tarball from ~209 kB to ~33 kB (99 files to 25, ~850 kB to ~101 kB unpacked). Nothing that was importable before has moved * note: no behavior change without `--shard`. A plain run writes the same files, prints the same output, and exits the same way as 1.4.0 Sharding needs three things right in the workflow: `fail-fast: false` on the diff --git a/README.md b/README.md index f40b6d8..c0b2495 100644 --- a/README.md +++ b/README.md @@ -335,9 +335,11 @@ summary print at all. refuses and names the gap rather than silently reporting 3 of 4 shards as a complete green run. - **Tests must register identically in every job.** Each shard fingerprints the - test list it discovered and `merge` verifies they match. Registering tests - conditionally — behind a feature flag, a date, `Math.random()` — makes the - fingerprints diverge and `merge` will say so. + ordered list of `"suite > test"` paths it discovered and `merge` verifies they + match. Registering tests conditionally — behind a feature flag, a date, + `Math.random()` — makes the fingerprints diverge and `merge` will say so. + (Paths rather than internal test ids: `twd-js` assigns those at registration + time and they differ on every page load, so each shard's browser sees its own.) - **`maxFailures` is per shard.** Four shards at the default of 10 can reach 40 failures between them before all four bail. - **`--test` and `--shard` compose:** filters resolve first, then the filtered list diff --git a/docs/superpowers/specs/2026-08-19-shardable-run-artifacts-design.md b/docs/superpowers/specs/2026-08-19-shardable-run-artifacts-design.md index ed0873b..3bbadb8 100644 --- a/docs/superpowers/specs/2026-08-19-shardable-run-artifacts-design.md +++ b/docs/superpowers/specs/2026-08-19-shardable-run-artifacts-design.md @@ -140,8 +140,20 @@ storing derived totals would let them drift out of agreement under merge. ### `discovery.fingerprint` is the safety net -The fingerprint is a hash of `{ orderedIds: , -filters: }`. +> **Correction (fixed in review, report schema v2).** This section originally +> specified the hash over test **ids**. That could never work: `twd-js` mints +> ids with `Math.random()` at registration time, so every page load — and +> therefore every shard's browser — invents different ids for the same tests. +> The hash was a per-load nonce and `merge` rejected every correct multi-shard +> run. Identity is now split in two: `tests[].path` (the `"suite > test"` +> string, resolved in the shard that ran the test) is what the fingerprint +> hashes and what the summary displays, and `tests[].index` (position in the +> discovered order) is the cross-shard identity key. The same defect made the +> "no test id appears in two reports" check below vacuous; it is keyed on +> `index` now. + +The fingerprint is a hash of `{ orderedPaths: test" path, in order>, filters: }`. Round-robin sharding is correct only if every job enumerates an identical test set. That silently breaks if the app registers tests conditionally — a feature @@ -254,7 +266,8 @@ Validation runs before anything is combined. All of these are fatal: - all `discovery.fingerprint` equal - all `shards[].total` equal, and `shards[].index` covers `1..total` exactly — no gaps, no duplicates -- no test id appears in two reports +- no test *position* (`tests[].index`) appears in two reports — see the + correction above; ids cannot serve as a key Combining is then mechanical. `tests`, `contracts.results` and `contracts.skipped` concat. `shards` concats sorted by index. `handlers` and @@ -339,7 +352,7 @@ nothing. Invalid shard specs error and exit 1. | Shard crashed before upload, artifact missing | error, names the missing index — never a silent 3-of-4 green | | Shard uploaded but bailed at `maxFailures` | merged report carries its `notRun`, contracts flagged `partial`, exit driven by the real failures | | Shards saw different test sets | error citing conditional registration or mismatched code | -| Same test id in two shards | error — shard math bug | +| Same test position in two shards | error — shard math bug | | Fewer tests than shards | valid: empty shard writes `tests: []` | | `--shard 5/4`, `0/4`, `abc` | parse error, exit 1 | | A shard's `coverage.json` absent | that shard does not contribute; merge states the contributor count | diff --git a/package.json b/package.json index 986998d..3649256 100644 --- a/package.json +++ b/package.json @@ -11,6 +11,7 @@ "bin", "src", "README.md", + "CHANGELOG.md", "LICENSE" ], "scripts": { diff --git a/src/index.js b/src/index.js index 837735d..db025a5 100644 --- a/src/index.js +++ b/src/index.js @@ -167,9 +167,11 @@ export async function runTests(options = {}) { // Resolve the ordered id list to run: the filter result, or all tests. // - // allTestIds is the full ordered list, before filtering or slicing. This is - // what the fingerprint hashes and what discovery.totalTests reports, so - // every shard agrees on it regardless of which slice it took. + // allTestIds is the full ordered list, before filtering or slicing. Its + // order is what the fingerprint covers (as paths — the ids themselves are + // random per page load) and its length is discovery.totalTests, so every + // shard agrees on both regardless of which slice it took. filteredIds is + // the list the shards divide, so it is what executed + notRun must total. const allTestIds = orderedTestIds(registeredHandlers); const filteredIds = selectedIds ?? allTestIds; const baseIds = sharded @@ -327,7 +329,11 @@ export async function runTests(options = {}) { }; if (contractsConfigured && (sharded || !stoppedEarly)) { - if (collectedMocks.size === 0) { + // Never under sharding. A shard whose slice exercised no mocks — and any + // shard with an empty slice — collects nothing, which is normal, so this + // would advertise a twd-js version problem that does not exist on the + // happy path of every sharded CI run. + if (collectedMocks.size === 0 && !sharded) { console.log('\nNo mocks collected — ensure twd-js supports contract collection'); } const validationOutput = validateMocks(collectedMocks, contractValidators); @@ -432,6 +438,7 @@ export async function runTests(options = {}) { startedAt, endedAt, allTestIds, + filteredIds, filters: testFilters, handlers, tests: testStatus, diff --git a/src/mergeCommand.js b/src/mergeCommand.js index be4e3a9..a929962 100644 --- a/src/mergeCommand.js +++ b/src/mergeCommand.js @@ -120,7 +120,7 @@ export function runMerge({ dir, out = null } = {}) { if (!totals.consistent) { console.warn( `Warning: shard totals do not add up — ${totals.executed} executed + ` + - `${totals.notRun} not run != ${merged.discovery.totalTests} discovered. ` + + `${totals.notRun} not run != ${totals.expected} selected. ` + 'This points at a shard-slicing bug, not at your tests.' ); } diff --git a/src/mergeReports.js b/src/mergeReports.js index 0fcee5f..a0e2e7d 100644 --- a/src/mergeReports.js +++ b/src/mergeReports.js @@ -1,3 +1,5 @@ +import { REPORT_SCHEMA_VERSION } from './runReport.js'; + /** * Combines shard reports into one report of the same shape. * @@ -23,6 +25,17 @@ export function mergeRunReports(reports) { ); } + // Agreement is not enough. Reports from a newer twd-cli agree with each other + // and would be merged by an older binary against a schema it does not + // understand — the exact silent mis-merge the field exists to prevent. + if (versions[0] !== REPORT_SCHEMA_VERSION) { + throw new Error( + `Shard reports use report schema v${versions[0]}, but this twd-cli reads ` + + `v${REPORT_SCHEMA_VERSION}. Every shard job and the merge job must run the ` + + 'same twd-cli version.' + ); + } + const fingerprints = new Set(reports.map((r) => r.discovery.fingerprint)); if (fingerprints.size > 1) { throw new Error( @@ -50,16 +63,25 @@ export function mergeRunReports(reports) { byIndex.add(shard.index); } + // Overlap is detected on tests[].index — the test's position in the discovered + // order — not on tests[].id. twd-js ids are Math.random() per page load, so + // every shard invents its own and an id-keyed check can never fire: it looked + // like a guard while proving nothing. Position is deterministic, so it is a + // real one. The path cannot be the key either: duplicate test names share a + // path and may legally land in different shards. const tests = []; - const testIds = new Set(); + const positions = new Set(); for (const report of reports) { for (const test of report.tests) { - if (testIds.has(test.id)) { - throw new Error( - `Test id "${test.id}" appears in more than one shard — the shard slices overlap.` - ); + if (test.index != null) { + if (positions.has(test.index)) { + throw new Error( + `Test "${test.path ?? test.id}" (position ${test.index}) appears in more ` + + 'than one shard — the shard slices overlap.' + ); + } + positions.add(test.index); } - testIds.add(test.id); tests.push(test); } } @@ -72,9 +94,10 @@ export function mergeRunReports(reports) { shards: [...shards].sort((a, b) => a.index - b.index), discovery: first.discovery, selection: first.selection, - // Invariant across shards in practice: every shard enumerates the same app. - // Not proven by the fingerprint, which covers the ordered test-id list and - // the filters, not handler metadata. Taking the first is the documented + // Only the first shard's map, and only its own ids resolve in it — twd-js + // ids are per-page-load random. That is why every test carries its own + // resolved `path` and renderers prefer it; this stays for the ids it can + // still explain and for diagnostics. Taking the first is the documented // contract, pinned by a test. handlers: first.handlers, tests, @@ -125,9 +148,16 @@ export function reportTimings(report) { export function reportTotals(report) { const executed = report.shards.reduce((sum, s) => sum + s.executed, 0); const notRun = report.shards.reduce((sum, s) => sum + s.notRun, 0); + // Against the count the shards divided, not everything discovered. With + // --test active those differ by every excluded test, and comparing to + // discovery.totalTests reported a slicing bug on a correct run. The fallback + // is unreachable for a v2 report; it only keeps a hand-built one from + // comparing against undefined and always warning. + const expected = report.selection?.selectedTests ?? report.discovery.totalTests; return { executed, notRun, - consistent: executed + notRun === report.discovery.totalTests, + expected, + consistent: executed + notRun === expected, }; } diff --git a/src/runReport.js b/src/runReport.js index ef043ec..2ae297f 100644 --- a/src/runReport.js +++ b/src/runReport.js @@ -1,9 +1,10 @@ import crypto from 'node:crypto'; +import { buildTestPath } from './buildTestPath.js'; -export const REPORT_SCHEMA_VERSION = 1; +export const REPORT_SCHEMA_VERSION = 2; /** - * Hash of the full ordered test id list plus any active --test filters. + * Hash of the full ordered list of test *paths* plus any active --test filters. * * Round-robin sharding is correct only if every job enumerates an identical * test set in an identical order. That breaks silently if the app registers @@ -11,11 +12,20 @@ export const REPORT_SCHEMA_VERSION = 1; * jobs did not build the same code: tests quietly never run and the build stays * green. Shards compare fingerprints at merge time so it becomes an error. * + * Paths, not ids. twd-js mints test ids with `Math.random()` at registration + * time, so the same test carries a different id on every page load — and every + * shard boots its own browser. Hashing ids made the fingerprint a per-load + * nonce that could never match, so `merge` rejected every correct multi-shard + * run. A `"suite > test"` path is derived from names, so it is stable across + * loads, and it is the same string `--test` already filters on. The check is + * also strictly stronger this way: a conditionally-registered test still + * changes the hash, because its path drops out of the ordered list. + * * Filters are OR'd, so their order carries no meaning and is normalized away. */ -export function fingerprintTests(orderedIds, filters = []) { +export function fingerprintTests(orderedPaths, filters = []) { const payload = JSON.stringify({ - orderedIds, + orderedPaths, filters: [...filters].sort(), }); const digest = crypto.createHash('sha256').update(payload).digest('hex'); @@ -31,12 +41,17 @@ export function fingerprintTests(orderedIds, filters = []) { * * `startedAt` and `endedAt` are epoch milliseconds; the report stores ISO * strings plus the derived duration. + * + * `allTestIds` is the whole discovered suite in order; `filteredIds` is what + * survived `--test` and therefore what the shards actually divide between them. + * They are the same list when no filter is active. */ export function buildRunReport({ shard, startedAt, endedAt, allTestIds, + filteredIds = null, filters = [], handlers, tests, @@ -47,6 +62,14 @@ export function buildRunReport({ recording = null, contracts = null, }) { + // Resolved here, in the shard that ran the tests, because this is the only + // place the handler map is valid: ids are per-page-load random, so shard 2's + // ids mean nothing in shard 1's handler map — and shard 1's map is the one a + // merged report keeps. + const orderedPaths = allTestIds.map((id) => buildTestPath(id, handlers)); + const positions = new Map(allTestIds.map((id, i) => [id, i])); + const selectedIds = filteredIds ?? allTestIds; + return { schemaVersion: REPORT_SCHEMA_VERSION, shards: [ @@ -68,11 +91,24 @@ export function buildRunReport({ ], discovery: { totalTests: allTestIds.length, - fingerprint: fingerprintTests(allTestIds, filters), + fingerprint: fingerprintTests(orderedPaths, filters), }, - selection: { filters: [...filters] }, + // selectedTests is the count the shards divided, which is what + // executed + notRun must add up to. Comparing against totalTests instead + // reports a slicing bug on any correct `--test` + `--shard` run. + selection: { filters: [...filters], selectedTests: selectedIds.length }, handlers, - tests, + tests: tests.map((test) => ({ + ...test, + // For display. May be null if the handler somehow went missing, so every + // renderer has to tolerate that. + path: buildTestPath(test.id, handlers), + // Identity. Position in the shard-independent ordered list is stable + // across shards because registration order is deterministic, unlike the + // random id. The path cannot serve as the key: two tests may share one + // (duplicate names) and can legally land in different shards. + index: positions.has(test.id) ? positions.get(test.id) : null, + })), contracts: contracts ?? { configured: false, partial: false, diff --git a/src/testSummary.js b/src/testSummary.js index c7d717c..f2eb971 100644 --- a/src/testSummary.js +++ b/src/testSummary.js @@ -1,5 +1,20 @@ import { buildTestPath } from './buildTestPath.js'; +/** + * Display name for one test result. + * + * `entry.path` is preferred because it was resolved inside the shard that ran + * the test, where the handler map was valid. A merged report keeps only the + * first shard's handlers, and twd-js ids are random per page load, so + * buildTestPath cannot resolve anything from shards 2..n — every failure would + * print a bare random id in the one place the merge exists to produce. The + * buildTestPath call stays for a live, non-sharded run, whose entries carry no + * path. + */ +function resolvePath(entry, handlers) { + return entry.path ?? buildTestPath(entry.id, handlers) ?? entry.id; +} + export function formatRunComplete({ testStatus, handlers, @@ -38,7 +53,7 @@ export function formatRunComplete({ if (failures.length > 0) { lines.push('', ` Failed tests (${failures.length}):`); for (const failure of failures) { - const testPath = buildTestPath(failure.id, handlers) ?? failure.id; + const testPath = resolvePath(failure, handlers); lines.push(` × ${testPath}`); if (failure.error) { lines.push(` ${String(failure.error).replace(/\n/g, '\n ')}`); @@ -50,7 +65,7 @@ export function formatRunComplete({ if (retried.length > 0) { lines.push('', ` Retried (${retried.length}):`); for (const t of retried) { - const testPath = buildTestPath(t.id, handlers) ?? t.id; + const testPath = resolvePath(t, handlers); lines.push(` ✓ ${testPath} (passed on attempt ${t.retryAttempt})`); } } diff --git a/tests/mergeCommand.test.js b/tests/mergeCommand.test.js index 2eb020f..e7f663a 100644 --- a/tests/mergeCommand.test.js +++ b/tests/mergeCommand.test.js @@ -13,6 +13,7 @@ import { loadConfig } from '../src/config.js'; import { printContractReport } from '../src/contractReport.js'; import { readShardReports, readShardCoverage } from '../src/reportFiles.js'; import { runMerge } from '../src/mergeCommand.js'; +import { REPORT_SCHEMA_VERSION } from '../src/runReport.js'; const HANDLERS = [ { id: 's1', name: 'Login', parent: null, type: 'suite' }, @@ -20,10 +21,19 @@ const HANDLERS = [ { id: 't2', name: 'b', parent: 's1', type: 'test' }, ]; +// Only shard 1's ids exist in HANDLERS, matching reality: twd-js mints ids with +// Math.random() at registration, so each shard's browser invents its own. Every +// entry carries the path its own shard resolved, plus its position in the +// discovered order as the cross-shard identity. function shardReport(index, overrides = {}) { - const { total = 2, tests = [{ id: `t${index}`, status: 'pass' }], failed = 0 } = overrides; + const { + total = 2, + tests = [{ id: index === 1 ? 't1' : `r${index}-x`, path: `Login > ${'ab'[index - 1]}`, index: index - 1, status: 'pass' }], + failed = 0, + selectedTests = 2, + } = overrides; return { - schemaVersion: 1, + schemaVersion: REPORT_SCHEMA_VERSION, shards: [{ index, total, startedAt: `2026-08-19T10:00:0${index}.000Z`, @@ -33,7 +43,7 @@ function shardReport(index, overrides = {}) { stoppedEarly: false, coverageFile: 'coverage.json', recording: null, }], discovery: { totalTests: 2, fingerprint: 'sha256:same' }, - selection: { filters: [] }, + selection: { filters: [], selectedTests }, handlers: HANDLERS, tests, contracts: { configured: false, partial: false, results: [], skipped: [] }, @@ -122,7 +132,7 @@ describe('runMerge', () => { const call = vi.mocked(fs.writeFileSync).mock.calls .find(([f]) => String(f).endsWith('merged-run.json')); const merged = JSON.parse(String(call[1])); - expect(merged.tests.map((t) => t.id)).toEqual(['t1', 't2']); + expect(merged.tests.map((t) => t.index)).toEqual([0, 1]); expect(merged.shards.map((s) => s.index)).toEqual([1, 2]); }); @@ -236,14 +246,11 @@ describe('runMerge', () => { expect(writtenFiles().some((f) => f.endsWith('contract-report.md'))).toBe(true); }); - // Executed + not-run has to account for every discovered test. When it does + // Executed + not-run has to account for every selected test. When it does // not, the shard math dropped tests on the floor and nothing else would say so. - it('warns when the shard totals do not account for every discovered test', () => { + it('warns when the shard totals do not account for every selected test', () => { const warn = vi.spyOn(console, 'warn'); - const wrongTotal = (i) => ({ - ...shardReport(i), - discovery: { totalTests: 3, fingerprint: 'sha256:same' }, - }); + const wrongTotal = (i) => shardReport(i, { selectedTests: 3 }); vi.mocked(readShardReports).mockReturnValue([ { dir: 'a', report: wrongTotal(1) }, { dir: 'b', report: wrongTotal(2) }, @@ -254,6 +261,27 @@ describe('runMerge', () => { expect(warn.mock.calls.flat().join('\n')).toMatch(/shard totals do not add up/); }); + // The false alarm this replaces: a --test filter narrows what the shards + // divide, so a correct filtered run has executed + notRun below + // discovery.totalTests and used to be reported as a shard-slicing bug. + it('does not warn when a filter narrowed what the shards divided', () => { + const warn = vi.spyOn(console, 'warn'); + const filtered = (i) => { + const report = shardReport(i); + report.discovery = { totalTests: 40, fingerprint: 'sha256:same' }; + report.selection = { filters: ['Login'], selectedTests: 2 }; + return report; + }; + vi.mocked(readShardReports).mockReturnValue([ + { dir: 'a', report: filtered(1) }, + { dir: 'b', report: filtered(2) }, + ]); + + runMerge({ dir: '.twd/shards' }); + + expect(warn.mock.calls.flat().join('\n')).not.toMatch(/do not add up/); + }); + it('prints the merged run-complete block with a shard breakdown', () => { const log = vi.spyOn(console, 'log'); vi.mocked(readShardReports).mockReturnValue([ @@ -267,4 +295,49 @@ describe('runMerge', () => { expect(output).toContain('--- Run complete ---'); expect(output).toContain('Shards: 1 ✓1 | 2 ✓1'); }); + + // The merged summary is the one output the merge exists to produce, and a + // failure from any shard but the first used to render as a raw random id: the + // merged report keeps only shard 1's handlers, which cannot resolve shard 2's + // ids. Each entry now carries the path its own shard resolved. + it('names failed tests from later shards instead of printing their raw ids', () => { + const log = vi.spyOn(console, 'log'); + vi.mocked(readShardReports).mockReturnValue([ + { dir: 'a', report: shardReport(1) }, + { + dir: 'b', + report: shardReport(2, { + failed: 1, + tests: [{ id: 'k3j2h1g9d', path: 'Login > b', index: 1, status: 'fail', error: 'boom' }], + }), + }, + ]); + + const hasFailures = runMerge({ dir: '.twd/shards' }); + + const output = log.mock.calls.flat().join('\n'); + expect(hasFailures).toBe(true); + expect(output).toContain('× Login > b'); + expect(output).not.toContain('k3j2h1g9d'); + }); + + // Retried tests render through the same path resolution. + it('names retried tests from later shards', () => { + const log = vi.spyOn(console, 'log'); + vi.mocked(readShardReports).mockReturnValue([ + { dir: 'a', report: shardReport(1) }, + { + dir: 'b', + report: shardReport(2, { + tests: [{ id: 'zzzflaky', path: 'Login > b', index: 1, status: 'pass', retryAttempt: 2 }], + }), + }, + ]); + + runMerge({ dir: '.twd/shards' }); + + const output = log.mock.calls.flat().join('\n'); + expect(output).toContain('✓ Login > b (passed on attempt 2)'); + expect(output).not.toContain('zzzflaky'); + }); }); diff --git a/tests/mergeReports.test.js b/tests/mergeReports.test.js index 8233967..33f3c89 100644 --- a/tests/mergeReports.test.js +++ b/tests/mergeReports.test.js @@ -5,6 +5,7 @@ import { reportTimings, reportTotals, } from '../src/mergeReports.js'; +import { REPORT_SCHEMA_VERSION } from '../src/runReport.js'; const HANDLERS = [ { id: 's1', name: 'Login', parent: null, type: 'suite' }, @@ -15,10 +16,13 @@ const HANDLERS = [ const FINGERPRINT = 'sha256:deadbeef'; +// Ids are per-page-load random in reality, so shard n's report carries ids +// nothing else can resolve. index (position in the discovered order) and path +// are what stay stable, and they are what the merge relies on. function makeReport(index, overrides = {}) { const { total = 3, - tests = [{ id: `t${index}`, status: 'pass' }], + tests = [{ id: `r${index}-t`, path: `Login > ${'abc'[index - 1]}`, index: index - 1, status: 'pass' }], startedAt = `2026-08-19T10:00:0${index}.000Z`, endedAt = `2026-08-19T10:00:1${index}.000Z`, durationMs = 10_000, @@ -29,8 +33,9 @@ function makeReport(index, overrides = {}) { coverageFile = 'coverage.json', contracts = { configured: true, partial: false, results: [], skipped: [] }, fingerprint = FINGERPRINT, - schemaVersion = 1, + schemaVersion = REPORT_SCHEMA_VERSION, totalTests = 3, + selectedTests = 3, } = overrides; return { @@ -40,7 +45,7 @@ function makeReport(index, overrides = {}) { executed, notRun, failed, stoppedEarly, coverageFile, recording: null, }], discovery: { totalTests, fingerprint }, - selection: { filters: [] }, + selection: { filters: [], selectedTests }, handlers: HANDLERS, tests, contracts, @@ -50,7 +55,8 @@ function makeReport(index, overrides = {}) { describe('mergeRunReports', () => { it('concatenates tests across shards', () => { const merged = mergeRunReports([makeReport(1), makeReport(2), makeReport(3)]); - expect(merged.tests.map((t) => t.id)).toEqual(['t1', 't2', 't3']); + expect(merged.tests.map((t) => t.index)).toEqual([0, 1, 2]); + expect(merged.tests.map((t) => t.path)).toEqual(['Login > a', 'Login > b', 'Login > c']); }); it('sorts shard descriptors by index regardless of input order', () => { @@ -81,7 +87,7 @@ describe('mergeRunReports', () => { const merged = mergeRunReports([first, second]); expect(merged.discovery.totalTests).toBe(3); - expect(merged.selection).toEqual({ filters: [] }); + expect(merged.selection).toEqual({ filters: [], selectedTests: 3 }); expect(merged.handlers).toEqual(HANDLERS); expect(merged.contracts.configured).toBe(true); }); @@ -123,8 +129,21 @@ describe('mergeRunReports', () => { }); it('throws when schema versions disagree', () => { - expect(() => mergeRunReports([makeReport(1), makeReport(2, { schemaVersion: 2 })])) - .toThrow(/schemaVersion/); + expect(() => mergeRunReports([ + makeReport(1), + makeReport(2, { schemaVersion: REPORT_SCHEMA_VERSION + 1 }), + ])).toThrow(/schemaVersion/); + }); + + // Agreeing with each other is not enough. Reports from a newer twd-cli agree, + // and merging them against a schema this binary does not know is the silent + // mis-merge the field exists to prevent. + it('throws when every shard agrees on a version this build does not read', () => { + const future = REPORT_SCHEMA_VERSION + 1; + expect(() => mergeRunReports([ + makeReport(1, { schemaVersion: future }), + makeReport(2, { schemaVersion: future }), + ])).toThrow(new RegExp(`schema v${future}, but this twd-cli reads v${REPORT_SCHEMA_VERSION}`)); }); // The safety net: shards that saw different test sets must never be combined. @@ -143,11 +162,23 @@ describe('mergeRunReports', () => { .toThrow(/more than once/); }); - it('throws when a test id appears in two shards', () => { + // Overlap is keyed on position, not id: shards never agree on ids, so the old + // id check could not fire at all. + it('throws when the same position appears in two shards', () => { expect(() => mergeRunReports([ - makeReport(1, { tests: [{ id: 'dup', status: 'pass' }] }), - makeReport(2, { tests: [{ id: 'dup', status: 'pass' }] }), - ])).toThrow(/"dup" appears in more than one shard/); + makeReport(1, { tests: [{ id: 'aaa', path: 'Login > a', index: 0, status: 'pass' }] }), + makeReport(2, { tests: [{ id: 'zzz', path: 'Login > a', index: 0, status: 'pass' }] }), + ])).toThrow(/"Login > a" \(position 0\) appears in more than one shard/); + }); + + // Two tests may legitimately share a "suite > test" path and land in + // different shards. Keying identity on the path would fail a correct run. + it('accepts duplicate paths in different shards when the positions differ', () => { + const merged = mergeRunReports([ + makeReport(1, { tests: [{ id: 'aaa', path: 'Login > a', index: 0, status: 'pass' }] }), + makeReport(2, { tests: [{ id: 'zzz', path: 'Login > a', index: 1, status: 'pass' }] }), + ]); + expect(merged.tests).toHaveLength(2); }); it('does not mutate the input reports', () => { @@ -187,12 +218,13 @@ describe('reportTimings', () => { }); describe('reportTotals', () => { - it('sums executed and notRun and confirms they account for discovery', () => { + it('sums executed and notRun and confirms they account for the selection', () => { const merged = mergeRunReports([makeReport(1), makeReport(2), makeReport(3)]); - expect(reportTotals(merged)).toEqual({ executed: 3, notRun: 0, consistent: true }); + expect(reportTotals(merged)) + .toEqual({ executed: 3, notRun: 0, expected: 3, consistent: true }); }); - it('flags totals that do not add up to the discovered count', () => { + it('flags totals that do not add up to the selected count', () => { const merged = mergeRunReports([makeReport(1), makeReport(2)]); expect(reportTotals(merged).consistent).toBe(false); }); @@ -203,6 +235,17 @@ describe('reportTotals', () => { makeReport(2, { executed: 1, notRun: 1, stoppedEarly: true, failed: 1 }), makeReport(3), ]); - expect(reportTotals(merged)).toEqual({ executed: 3, notRun: 1, consistent: false }); + expect(reportTotals(merged)) + .toEqual({ executed: 3, notRun: 1, expected: 3, consistent: false }); + }); + + // The regression this fixes: --test narrows what the shards divide, so + // comparing against every discovered test called a correct run a slicing bug. + it('measures a filtered run against the filtered count, not the whole suite', () => { + const merged = mergeRunReports([ + makeReport(1, { totalTests: 40, selectedTests: 2, tests: [] }), + makeReport(2, { totalTests: 40, selectedTests: 2, tests: [] }), + ]); + expect(reportTotals(merged)).toEqual({ executed: 2, notRun: 0, expected: 2, consistent: true }); }); }); diff --git a/tests/runReport.test.js b/tests/runReport.test.js index ba57eb8..67c974e 100644 --- a/tests/runReport.test.js +++ b/tests/runReport.test.js @@ -4,8 +4,12 @@ import { buildRunReport, fingerprintTests, REPORT_SCHEMA_VERSION } from '../src/ const handlers = [ { id: 's1', name: 'Login', parent: null, type: 'suite' }, { id: 't1', name: 'works', parent: 's1', type: 'test' }, + { id: 't2', name: 'also works', parent: 's1', type: 'test' }, + { id: 't3', name: 'still works', parent: 's1', type: 'test' }, ]; +const PATHS = ['Login > works', 'Login > also works', 'Login > still works']; + function build(overrides = {}) { return buildRunReport({ shard: { index: 2, total: 4 }, @@ -33,11 +37,11 @@ describe('fingerprintTests', () => { // Order matters: round-robin slicing is only correct if every shard sees the // same list in the same order. - it('changes when the id order changes', () => { + it('changes when the path order changes', () => { expect(fingerprintTests(['a', 'b'])).not.toBe(fingerprintTests(['b', 'a'])); }); - it('changes when the id set changes', () => { + it('changes when the path set changes', () => { expect(fingerprintTests(['a', 'b'])).not.toBe(fingerprintTests(['a', 'b', 'c'])); }); @@ -88,13 +92,50 @@ describe('buildRunReport', () => { it('records total discovered tests and the fingerprint', () => { const report = build(); expect(report.discovery.totalTests).toBe(3); - expect(report.discovery.fingerprint).toBe(fingerprintTests(['t1', 't2', 't3'], [])); + expect(report.discovery.fingerprint).toBe(fingerprintTests(PATHS, [])); + }); + + // The whole point of the path-based fingerprint: twd-js ids are Math.random() + // per page load, so two shards of the same suite never agree on ids. If the + // fingerprint were keyed on them, merge would reject every correct run. + it('fingerprints the same suite identically when the ids differ', () => { + const shardTwoHandlers = handlers.map((h) => ({ + ...h, + id: `x${h.id}`, + parent: h.parent ? `x${h.parent}` : h.parent, + })); + const shardTwo = build({ + allTestIds: ['xt1', 'xt2', 'xt3'], + handlers: shardTwoHandlers, + tests: [{ id: 'xt1', status: 'pass' }], + }); + expect(shardTwo.discovery.fingerprint).toBe(build().discovery.fingerprint); }); - it('carries handlers and tests through untouched', () => { - const report = build(); - expect(report.handlers).toEqual(handlers); - expect(report.tests).toEqual([{ id: 't1', status: 'pass' }]); + it('carries handlers through untouched', () => { + expect(build().handlers).toEqual(handlers); + }); + + // path is for display, index is for identity. See buildRunReport. + it('stamps each test with its resolved path and its position in the order', () => { + expect(build({ tests: [{ id: 't3', status: 'fail', error: 'boom' }] }).tests).toEqual([ + { id: 't3', status: 'fail', error: 'boom', path: 'Login > still works', index: 2 }, + ]); + }); + + // Positions are shard-independent, so a merged report can detect a genuine + // overlap with them where random ids proved nothing. + it('numbers positions from the full ordered list, not the shard slice', () => { + const report = build({ + tests: [{ id: 't2', status: 'pass' }, { id: 't3', status: 'pass' }], + }); + expect(report.tests.map((t) => t.index)).toEqual([1, 2]); + }); + + // Renderers fall back on a null path; inventing one would be worse. + it('leaves path null and index null when the handler is missing', () => { + const report = build({ tests: [{ id: 'ghost', status: 'pass' }] }); + expect(report.tests[0]).toEqual({ id: 'ghost', status: 'pass', path: null, index: null }); }); it('copies the filters rather than aliasing them', () => { @@ -104,6 +145,18 @@ describe('buildRunReport', () => { expect(report.selection.filters).toEqual(['Login']); }); + // What executed + notRun has to add up to. With --test active this is smaller + // than discovery.totalTests, and comparing against the latter reported a + // shard-slicing bug on every correct filtered run. + it('records the filtered count the shards divided', () => { + expect(build({ filteredIds: ['t2', 't3'], filters: ['works'] }).selection.selectedTests) + .toBe(2); + }); + + it('falls back to the whole suite when no filter is active', () => { + expect(build().selection.selectedTests).toBe(3); + }); + it('defaults contracts to an unconfigured empty block', () => { expect(build().contracts).toEqual({ configured: false, partial: false, results: [], skipped: [], diff --git a/tests/runTests.test.js b/tests/runTests.test.js index a774316..547af64 100644 --- a/tests/runTests.test.js +++ b/tests/runTests.test.js @@ -1,5 +1,6 @@ import { describe, it, expect, vi, beforeEach, afterEach } from "vitest"; import { runTests } from "../src/index.js"; +import { REPORT_SCHEMA_VERSION } from "../src/runReport.js"; vi.mock('fs'); vi.mock('puppeteer'); @@ -1239,12 +1240,51 @@ describe('runTests sharding', () => { expect(fs.mkdirSync).toHaveBeenCalledWith('./.twd/run', { recursive: true }); const report = runJson(); - expect(report.schemaVersion).toBe(1); + expect(report.schemaVersion).toBe(REPORT_SCHEMA_VERSION); expect(report.shards[0]).toMatchObject({ index: 2, total: 2, executed: 2 }); expect(report.discovery.totalTests).toBe(4); expect(report.tests.map((t) => t.id)).toEqual(['2', '4']); }); + // path is resolved here, in the shard that ran the test, because twd-js ids + // are random per page load: shard 2's ids do not exist in the handler map a + // merged report keeps, so the merged summary could not name these tests + // otherwise. index is the cross-shard identity — the same number in every + // shard, unlike the id. + it('stamps each result with its resolved path and its position in the order', async () => { + const { handlers, testStatus } = fourTests(); + const page = createMockPage({ handlers, testStatus }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + + await runTests({ shard: { index: 2, total: 2 } }); + + expect(runJson().tests).toEqual([ + { id: '2', status: 'pass', path: 'b', index: 1 }, + { id: '4', status: 'pass', path: 'd', index: 3 }, + ]); + }); + + // The fingerprint is over paths, not ids, so two shards of the same suite + // agree on it even though their browsers minted entirely different ids. With + // ids it could never match and merge rejected every correct sharded run. + it('fingerprints identically across shards whose ids differ', async () => { + const { handlers, testStatus } = fourTests(); + const first = createMockPage({ handlers, testStatus }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(first)); + await runTests({ shard: { index: 1, total: 2 } }); + const shardOne = runJson().discovery.fingerprint; + + vi.clearAllMocks(); + vi.mocked(loadConfig).mockReturnValue({ ...defaultMockConfig }); + // Same suite, same order, all-new ids — exactly what a second browser does. + const relabeled = handlers.map((h) => ({ ...h, id: `x${h.id}` })); + const second = createMockPage({ handlers: relabeled, testStatus: [] }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(second)); + await runTests({ shard: { index: 2, total: 2 } }); + + expect(runJson().discovery.fingerprint).toBe(shardOne); + }); + it('honors --report-dir', async () => { const { handlers, testStatus } = fourTests(); const page = createMockPage({ handlers, testStatus }); @@ -1302,6 +1342,23 @@ describe('runTests sharding', () => { expect(runJson().shards[0].coverageFile).toBeNull(); expect(runJson().selection.filters).toEqual(['a']); }); + + // Filters resolve first, so the shards divide the filtered list, and that is + // the count executed + notRun has to add up to. Recording the unfiltered + // total instead made merge print a shard-slicing warning on a correct run. + it('records the filtered count as the selection the shards divided', async () => { + const { handlers } = fourTests(); + const page = createMockPage({ handlers, testStatus: [{ id: '1', status: 'pass' }] }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + + await runTests({ shard: { index: 1, total: 2 }, testFilters: ['a'] }); + + const report = runJson(); + expect(report.discovery.totalTests).toBe(4); + expect(report.selection.selectedTests).toBe(1); + expect(report.shards[0].executed + report.shards[0].notRun) + .toBe(report.selection.selectedTests); + }); }); describe('runTests non-regression: non-sharded behavior is unchanged', () => { @@ -1362,6 +1419,26 @@ describe('runTests non-regression: non-sharded behavior is unchanged', () => { expect(validateMocks).not.toHaveBeenCalled(); }); + // The twd-js version hint gained a `&& !sharded` gate. A plain run with + // contracts configured and nothing collected really may be running a twd-js + // that cannot collect, so it must still say so. + it('still hints at twd-js when a non-sharded run collects no mocks', async () => { + const log = vi.spyOn(console, 'log'); + const handlers = [{ id: '1', name: 'a', type: 'test' }]; + const page = createMockPage({ handlers, testStatus: [{ id: '1', status: 'pass' }] }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + vi.mocked(loadConfig).mockReturnValue({ + ...defaultMockConfig, contracts: [{ source: 'api.json' }], + }); + vi.mocked(loadContracts).mockResolvedValue([]); + vi.mocked(validateMocks).mockReturnValue({ results: [], skipped: [] }); + vi.mocked(printContractReport).mockReturnValue(false); + + await runTests(); + + expect(log.mock.calls.flat().join('\n')).toContain('No mocks collected'); + }); + // The markdown report gained a `&& !sharded` gate. A plain run must still // write it, and this is the only test that executes that block at all. it('still writes the contract markdown report on a non-sharded run', async () => { @@ -1434,6 +1511,29 @@ describe('runTests sharded behavior changes', () => { expect(files.some((f) => f.endsWith('run.json'))).toBe(true); }); + // A shard whose slice exercised no mocks — and any shard with an empty slice — + // collects nothing, which is normal. Printing the twd-js version hint there + // advertises a problem that does not exist, on the happy path of every + // sharded CI run. + it('does not hint at twd-js when a sharded run collects no mocks', async () => { + const log = vi.spyOn(console, 'log'); + const handlers = [{ id: '1', name: 'a', type: 'test' }]; + const page = createMockPage({ handlers, testStatus: [{ id: '1', status: 'pass' }] }); + vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); + vi.mocked(loadConfig).mockReturnValue({ + ...defaultMockConfig, contracts: [{ source: 'api.json' }], + }); + vi.mocked(loadContracts).mockResolvedValue([]); + vi.mocked(validateMocks).mockReturnValue({ results: [], skipped: [] }); + vi.mocked(printContractReport).mockReturnValue(false); + + await runTests({ shard: { index: 1, total: 2 } }); + + expect(log.mock.calls.flat().join('\n')).not.toContain('No mocks collected'); + // Still validated what it did collect — only the hint is suppressed. + expect(validateMocks).toHaveBeenCalled(); + }); + it('validates contracts on a sharded early stop and marks them partial', async () => { const handlers = [{ id: '1', name: 'a', type: 'test' }]; const page = createMockPage({ handlers, testStatus: [{ id: '1', status: 'fail', error: 'boom' }] }); diff --git a/tests/testSummary.test.js b/tests/testSummary.test.js index 98a17a1..b8a45a2 100644 --- a/tests/testSummary.test.js +++ b/tests/testSummary.test.js @@ -25,6 +25,41 @@ describe('formatRunComplete', () => { ); }); + // A merged report keeps only the first shard's handlers, and twd-js ids are + // random per page load, so a later shard's failure cannot be resolved from + // them — it used to print as a raw id in the merged summary. Each entry now + // carries the path its own shard resolved. + it('prefers the path the shard resolved over its own handler lookup', () => { + const block = formatRunComplete({ + testStatus: [ + { id: 'k3j2h1g9d', path: 'Cart > removes an item', status: 'fail', error: 'boom' }, + { id: 'z9y8x7w6v', path: 'Cart > applies a coupon', status: 'pass', retryAttempt: 2 }, + ], + handlers, + durationMs: 1000, + }); + expect(block).toContain('× Cart > removes an item'); + expect(block).toContain('✓ Cart > applies a coupon (passed on attempt 2)'); + expect(block).not.toContain('k3j2h1g9d'); + expect(block).not.toContain('z9y8x7w6v'); + }); + + // A live non-sharded run carries no path, and a null path is a legal value. + it('falls back to the handler lookup, then the raw id', () => { + const block = formatRunComplete({ + testStatus: [ + { id: 't1', status: 'fail', error: 'a' }, + { id: 't2', path: null, status: 'fail', error: 'b' }, + { id: 'ghost', status: 'fail', error: 'c' }, + ], + handlers, + durationMs: 1000, + }); + expect(block).toContain('× Login > shows error on wrong password'); + expect(block).toContain('× Login > redirects on success'); + expect(block).toContain('× ghost'); + }); + it('counts skipped tests', () => { const block = formatRunComplete({ testStatus: [ From ce29f5169a8f886c4653881f8855e13606a86c91 Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Thu, 20 Aug 2026 10:45:46 +0200 Subject: [PATCH 21/26] docs: sync the spec's schema block to v2, and pin the version literal MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two follow-ups parked during the review. The design doc's canonical schema block still described v1 — schemaVersion 1, no selection.selectedTests, and tests[] without path or index — while a correction note 46 lines below explained that identity had moved to paths and positions. Anyone reading top-to-bottom got the broken shape first. The block now matches what buildRunReport actually emits. Every schemaVersion assertion in the suite derived from REPORT_SCHEMA_VERSION, so editing the constant left all 438 tests green. That is self-defeating for a value whose only job is to reject shards produced by mismatched twd-cli builds. One literal assertion now pins it; verified by flipping the constant to 3 and watching only that test fail. Co-Authored-By: Claude Opus 5 (1M context) --- ...2026-08-19-shardable-run-artifacts-design.md | 17 ++++++++++++----- tests/runReport.test.js | 8 ++++++++ 2 files changed, 20 insertions(+), 5 deletions(-) diff --git a/docs/superpowers/specs/2026-08-19-shardable-run-artifacts-design.md b/docs/superpowers/specs/2026-08-19-shardable-run-artifacts-design.md index 3bbadb8..ffa0d40 100644 --- a/docs/superpowers/specs/2026-08-19-shardable-run-artifacts-design.md +++ b/docs/superpowers/specs/2026-08-19-shardable-run-artifacts-design.md @@ -94,7 +94,7 @@ both, and makes a normal run the N=1 case with no second code path. ```jsonc { - "schemaVersion": 1, + "schemaVersion": 2, "shards": [ { "index": 2, "total": 4, "startedAt": "2026-08-19T10:00:00.000Z", @@ -105,15 +105,17 @@ both, and makes a normal run the N=1 case with no second code path. "recording": { "file": "login.mp4", "bytes": 481920 } } ], "discovery": { "totalTests": 120, "fingerprint": "sha256:abc123..." }, - "selection": { "filters": [] }, + "selection": { "filters": [], "selectedTests": 120 }, "handlers": [ { "id": "...", "name": "...", "parent": "...", "type": "test" } ], - "tests": [ { "id": "...", "status": "pass", "retryAttempt": 2 } ], + "tests": [ { "id": "...", "status": "pass", "retryAttempt": 2, + "path": "Login > shows error", "index": 7 } ], "contracts": { "configured": true, "partial": false, "results": [], "skipped": [] } } ``` -`handlers` and `tests` reuse the exact shapes already flowing through -`src/index.js:128` and `:226`, so `buildTestPath`, `formatRunComplete` and +`handlers` passes through exactly as enumerated. `tests` is the in-page status +array plus two fields the shard resolves before writing — `path` for display and +`index` for identity (see the correction below) — so `formatRunComplete` and `generateContractMarkdown` need no data massaging. `contracts` is `validateMocks()`'s return value verbatim plus two flags. @@ -126,6 +128,11 @@ shard went red is most of that line's value. filters resolve first, then the filtered list is sharded. There is no companion `mode` field — a report only exists under `--shard`, so it would be a constant. +`selection.selectedTests` is the count the shards actually divided, which is what +`executed + notRun` must sum to. `discovery.totalTests` counts the whole suite, +so comparing against *that* reports a phantom slicing bug on any correct +`--test` + `--shard` run. + Coverage is **referenced, not embedded** — `coverageFile` names a sibling file. This keeps `run.json` readable by eye and keeps coverage in stock Istanbul format so `nyc` tooling works on it untouched. A coverage blob is routinely diff --git a/tests/runReport.test.js b/tests/runReport.test.js index 67c974e..0a68ae0 100644 --- a/tests/runReport.test.js +++ b/tests/runReport.test.js @@ -59,6 +59,14 @@ describe('fingerprintTests', () => { }); describe('buildRunReport', () => { + // Pins the literal on purpose. Every other assertion in the suite derives + // from REPORT_SCHEMA_VERSION, so editing the constant would otherwise leave + // all 438 tests green — and the version's only job is to be a tripwire for + // shards produced by mismatched twd-cli builds. + it('is at schema version 2', () => { + expect(REPORT_SCHEMA_VERSION).toBe(2); + }); + it('stamps the schema version', () => { expect(build().schemaVersion).toBe(REPORT_SCHEMA_VERSION); }); From 7936a96b10862acb5536a6d4485a9832ef77b472 Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Thu, 20 Aug 2026 19:10:32 +0200 Subject: [PATCH 22/26] perf(cli): stop merge from loading puppeteer MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit bin/twd-cli.js imported src/index.js at the top level, so `twd-cli merge` pulled in puppeteer and openapi-mock-validator before it even read argv — for a command whose own import graph needs only fs, path, node:crypto and istanbul-lib-coverage. runTests and runMerge are now imported inside their branches. Verified by removing node_modules/puppeteer entirely: `merge` still runs and exits 1 with its usual message, while `run` fails loudly with "Cannot find package 'puppeteer'". Co-Authored-By: Claude Opus 5 (1M context) --- bin/twd-cli.js | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/bin/twd-cli.js b/bin/twd-cli.js index fb08ffb..62c097d 100755 --- a/bin/twd-cli.js +++ b/bin/twd-cli.js @@ -1,14 +1,17 @@ #!/usr/bin/env node -import { runTests } from '../src/index.js'; +// runTests and runMerge are imported inside their branches, not here. A static +// import of src/index.js pulls in puppeteer, so `twd-cli merge` — which never +// opens a browser — would otherwise load the whole browser-automation graph +// before it even looked at argv. import { parseRunArgs, parseMergeArgs } from '../src/parseArgs.js'; -import { runMerge } from '../src/mergeCommand.js'; const command = process.argv[2]; if (command === 'run') { try { const { testFilters, record, shard, reportDir } = parseRunArgs(process.argv.slice(3)); + const { runTests } = await import('../src/index.js'); const hasFailures = await runTests({ testFilters, recordOverrides: record, @@ -25,6 +28,7 @@ if (command === 'run') { } else if (command === 'merge') { try { const { dir, out } = parseMergeArgs(process.argv.slice(3)); + const { runMerge } = await import('../src/mergeCommand.js'); const hasFailures = runMerge({ dir, out }); process.exit(hasFailures ? 1 : 0); } catch (error) { From 84fe76eac2967dfd053c1361ebf5f3ab982d75c8 Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Thu, 20 Aug 2026 19:10:32 +0200 Subject: [PATCH 23/26] feat(action): add a shard input to the composite action MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Anyone using BRIKEV/twd-cli/.github/actions/run could not shard: the action only ever ran `npx twd-cli run`, so sharded workflows had to inline every step themselves. It now takes `shard` (as /), plus `report-dir` and `upload-report`. The artifact upload is included and carries `if: always()`, since a red shard that uploads nothing leaves merge unable to tell "this shard failed" from "this shard never ran" — the single easiest thing to get wrong when wiring this by hand. The contract PR comment now skips itself with a notice when `shard` is set. A sharded run deliberately writes no contract markdown per shard, so that step belongs in the job that runs `twd-cli merge`. Co-Authored-By: Claude Opus 5 (1M context) --- .github/actions/run/action.yml | 70 +++++++++++++++++++++++++++++++++- 1 file changed, 68 insertions(+), 2 deletions(-) diff --git a/.github/actions/run/action.yml b/.github/actions/run/action.yml index 1fbd3ea..b159cb3 100644 --- a/.github/actions/run/action.yml +++ b/.github/actions/run/action.yml @@ -1,5 +1,5 @@ name: 'TWD CLI Run' -description: 'Run TWD browser tests with Puppeteer, including optional contract validation reporting' +description: 'Run TWD browser tests with Puppeteer, optionally as one shard of a parallel matrix, including contract validation reporting' branding: icon: 'check-circle' color: 'green' @@ -13,6 +13,25 @@ inputs: description: 'Post contract validation report as a PR comment (requires pull-requests: write permission)' required: false default: 'false' + shard: + description: >- + Run one shard of the suite, as / (e.g. 2/4). Each shard + discovers the whole suite and takes every nth test, so the test count + never has to be known in advance. Leave empty to run everything in one + job. Join the shards afterwards with `npx twd-cli merge`. + required: false + default: '' + report-dir: + description: 'Where the shard report is written. Only used when `shard` is set.' + required: false + default: '.twd/run' + upload-report: + description: >- + Upload the shard report as an artifact named `twd-run-`, which is + the layout `twd-cli merge` expects after actions/download-artifact. Only + used when `shard` is set. + required: false + default: 'true' runs: using: 'composite' @@ -29,10 +48,48 @@ runs: shell: bash run: npx puppeteer browsers install chrome + - name: Resolve shard + id: shard + shell: bash + env: + SHARD: ${{ inputs.shard }} + run: | + if [ -z "$SHARD" ]; then + echo "index=" >> "$GITHUB_OUTPUT" + exit 0 + fi + # Artifact names cannot contain "/", so the index alone identifies the + # shard. twd-cli itself validates the spec and fails loudly on a bad + # one, so this only has to be good enough to name the artifact. + if ! printf '%s' "$SHARD" | grep -qE '^[0-9]+/[0-9]+$'; then + echo "::error::Invalid shard \"$SHARD\". Expected /, e.g. 2/4." + exit 1 + fi + echo "index=${SHARD%%/*}" >> "$GITHUB_OUTPUT" + - name: Run TWD tests shell: bash working-directory: ${{ inputs.working-directory }} - run: npx twd-cli run + env: + SHARD: ${{ inputs.shard }} + REPORT_DIR: ${{ inputs.report-dir }} + run: | + if [ -n "$SHARD" ]; then + npx twd-cli run --shard "$SHARD" --report-dir "$REPORT_DIR" + else + npx twd-cli run + fi + + - name: Upload shard report + # always(): a red shard must still upload, or `merge` cannot tell "this + # shard failed" from "this shard never ran", and it reports the gap as a + # missing artifact instead of the real failure. + if: always() && inputs.shard != '' && inputs.upload-report == 'true' + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 + with: + name: twd-run-${{ steps.shard.outputs.index }} + path: ${{ inputs.working-directory }}/${{ inputs.report-dir }} + if-no-files-found: error - name: Post contract report to PR if: inputs.contract-report == 'true' && github.event_name == 'pull_request' @@ -42,7 +99,16 @@ runs: GH_TOKEN: ${{ github.token }} PR_NUMBER: ${{ github.event.pull_request.number }} RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + SHARD: ${{ inputs.shard }} run: | + # A sharded run deliberately writes no contract markdown: every shard + # would overwrite the others with a fraction of the mocks. `twd-cli + # merge` writes it, so the comment belongs in the merge job. + if [ -n "$SHARD" ]; then + echo "::notice::Skipping the contract PR comment: this is shard $SHARD. Post it from the job that runs \`twd-cli merge\`, which is what writes the report." + exit 0 + fi + REPORT_PATH=$(node -e " const fs = require('fs'); try { From f74cdd7eccf4657c043bb8ae8f1625f5ab885c2a Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Thu, 20 Aug 2026 19:10:32 +0200 Subject: [PATCH 24/26] docs: move sharding into docs/ and put contract validation first MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Contract validation is the more important feature and now comes before sharding in both the table of contents and the body. Sharding was 85 lines of a 445-line README, so it moves to docs/sharding.md behind a short summary; the README is down to 384 lines. The doc also gains what the README never said: sharding only pays once test time dominates per-job setup, with the break-even and measured numbers from a 256-test suite, plus the note that a short suite comes out slower. That omission was a trap — this project's own 71-test suite goes from 25s to 41s when sharded. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 112 ++++++++-------------------------- docs/sharding.md | 154 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 179 insertions(+), 87 deletions(-) create mode 100644 docs/sharding.md diff --git a/README.md b/README.md index c0b2495..6e4e0e6 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ CI/CD runner for [TWD (Test while developing)](https://brikev.github.io/twd/) - [Recording](#recording): capture a run to video, paced so it is watchable - [Contract Validation](#contract-validation): check your mocks against OpenAPI specs - [CI/CD Integration](#cicd-integration): GitHub Action and custom setups -- [Sharding across CI jobs](#sharding-across-ci-jobs): split a run across parallel CI jobs +- [Sharding across CI jobs](#sharding-across-ci-jobs): split a long run across parallel jobs ([details](docs/sharding.md)) - [How It Works](#how-it-works) - [Requirements](#requirements) @@ -268,92 +268,6 @@ jobs: run: npm run collect:coverage:text ``` -## Sharding across CI jobs - -A single run walks the whole suite in one browser. `--shard` splits it across -parallel CI jobs instead. - -```bash -npx twd-cli run --shard 2/4 # "I am job 2 of 4" -npx twd-cli merge .twd/shards # join the reports back together -``` - -The `4` is how many jobs you are running, **not** how many tests exist. You never -need to know the test count: each shard boots its own browser, discovers the whole -suite exactly as a normal run does, and keeps every 4th test. Add tests and the -same 4 jobs just split more of them. - -Each shard writes `run.json` and `coverage.json` to `./.twd/run` (change it with -`--report-dir`). `merge` reads the downloaded shard directories, combines the test -results, coverage and contract validation, prints one summary, and exits non-zero -if anything failed anywhere. - -```yaml -jobs: - test: - strategy: - fail-fast: false # or one red shard cancels the rest - matrix: - shard: [1, 2, 3, 4] - steps: - # ...checkout, npm ci, chrome, dev server... - - run: npx twd-cli run --shard ${{ matrix.shard }}/4 - - uses: actions/upload-artifact@v4 - if: always() # a red shard must still upload - with: - name: twd-run-${{ matrix.shard }} - path: .twd/run - - merge: - needs: [test] - if: ${{ !cancelled() }} # runs even though a shard went red - steps: - - uses: actions/checkout@v5 - - uses: actions/setup-node@v5 - with: - node-version: 24 - cache: npm - - run: npm ci - - uses: actions/download-artifact@v4 - with: - pattern: twd-run-* - path: .twd/shards - - run: npx twd-cli merge .twd/shards -``` - -Those three conditions are easy to miss and each one breaks the run: -`fail-fast: false` stops a red shard cancelling its siblings, `if: always()` on -upload keeps a red shard's report, and `if: ${{ !cancelled() }}` on merge lets the -summary print at all. - -### Notes - -- **Coverage.** Each shard writes its own `coverage.json`; `merge` combines them - into `.nyc_output/out.json` — but only when the whole run is green, matching how - a single run behaves. `merge` reports how many shards contributed. -- **Missing shards are an error.** If a shard job dies before uploading, `merge` - refuses and names the gap rather than silently reporting 3 of 4 shards as a - complete green run. -- **Tests must register identically in every job.** Each shard fingerprints the - ordered list of `"suite > test"` paths it discovered and `merge` verifies they - match. Registering tests conditionally — behind a feature flag, a date, - `Math.random()` — makes the fingerprints diverge and `merge` will say so. - (Paths rather than internal test ids: `twd-js` assigns those at registration - time and they differ on every page load, so each shard's browser sees its own.) -- **`maxFailures` is per shard.** Four shards at the default of 10 can reach 40 - failures between them before all four bail. -- **`--test` and `--shard` compose:** filters resolve first, then the filtered list - is sharded. As with any filtered run, coverage is skipped. -- **Recording** produces one clip per shard; they are not concatenated. -- **A missing shard leaves no merged report on disk.** `merge` throws before it - writes `.twd/merged-run.json`, so a CI step that uploads that path with - `if: always()` will find nothing when a shard is missing. The error message on - stderr is the diagnosis in that case. -- **`record.filename` collides under sharding.** Only the *derived* recording - filename is per-shard. If `record.filename` is set explicitly in - `twd.config.json`, every shard writes to the same video path. Use the derived - name, or a per-shard `--record-dir`, when recording a sharded run. - ## Contract Validation Validate your test mocks against OpenAPI specs to catch drift between your mocks and the real API. When a mock response doesn't match the spec, you'll see errors like: @@ -436,6 +350,30 @@ When `contractReportPath` is set and you use the action with `contract-report: ' Failed validations are included in a collapsible details section with a link to the full CI log. +## Sharding across CI jobs + +Long suites can be split across parallel CI jobs. Each shard runs one slice of +the suite and writes a report; `twd-cli merge` joins them into a single summary +and owns the exit code. + +```bash +npx twd-cli run --shard 2/4 # "I am job 2 of 4" +npx twd-cli merge .twd/shards # join the reports back together +``` + +The `4` is how many jobs you are running, **not** how many tests exist — each +shard discovers the whole suite itself and keeps every 4th test, so the suite can +grow without a workflow edit. + +**Sharding only pays on long suites.** It trades fixed per-job setup for parallel +execution, so a suite that runs in seconds comes out *slower*. As a rule of +thumb, two shards win once test time is more than twice the merge job's cost. + +See **[docs/sharding.md](docs/sharding.md)** for the full workflow, the three +conditions that are easy to get wrong, the break-even maths with measured +numbers, and the caveats — test independence, `maxFailures` being per shard, and +coverage on a red run. + ## Requirements - Node.js >= 20.19.x diff --git a/docs/sharding.md b/docs/sharding.md new file mode 100644 index 0000000..17dbe80 --- /dev/null +++ b/docs/sharding.md @@ -0,0 +1,154 @@ +# Sharding across CI jobs + +A single run walks the whole suite in one browser. `--shard` splits it across +parallel CI jobs instead. + +```bash +npx twd-cli run --shard 2/4 # "I am job 2 of 4" +npx twd-cli merge .twd/shards # join the reports back together +``` + +The `4` is how many jobs you are running, **not** how many tests exist. You never +need to know the test count: each shard boots its own browser, discovers the whole +suite exactly as a normal run does, and keeps every 4th test. Add tests and the +same 4 jobs just split more of them. + +Each shard writes `run.json` and `coverage.json` to `./.twd/run` (change it with +`--report-dir`). `merge` reads the downloaded shard directories, combines the test +results, coverage and contract validation, prints one summary, and exits non-zero +if anything failed anywhere. + +```yaml +jobs: + test: + strategy: + fail-fast: false # or one red shard cancels the rest + matrix: + shard: [1, 2, 3, 4] + steps: + # ...checkout, npm ci, chrome, dev server... + - run: npx twd-cli run --shard ${{ matrix.shard }}/4 + - uses: actions/upload-artifact@v4 + if: always() # a red shard must still upload + with: + name: twd-run-${{ matrix.shard }} + path: .twd/run + + merge: + needs: [test] + if: ${{ !cancelled() }} # runs even though a shard went red + steps: + - uses: actions/checkout@v5 + - uses: actions/setup-node@v5 + with: + node-version: 24 + cache: npm + - run: npm ci + - uses: actions/download-artifact@v4 + with: + pattern: twd-run-* + path: .twd/shards + - run: npx twd-cli merge .twd/shards +``` + +Those three conditions are easy to miss and each one breaks the run: +`fail-fast: false` stops a red shard cancelling its siblings, `if: always()` on +upload keeps a red shard's report, and `if: ${{ !cancelled() }}` on merge lets the +summary print at all. + +## When sharding pays + +Sharding trades fixed per-job setup for parallel test execution, so it only wins +once test time dominates. With per-job overhead `V`, total test time `T`, and a +merge job costing `M`, the wall clock goes from `V + T` to `V + T/N + M`. So `N` +shards help only when: + +``` +M < T (1 - 1/N) → for two shards, roughly T > 2M +``` + +Measured on a real suite of 256 browser tests, where `V` was ~115s and the merge +job ~126s (89s of which was a SonarCloud scan): + +| Shards | Wall clock | Runner time | +|---|---|---| +| 1 | 12.6 min | baseline | +| 2 | ~8.1 min | +15% | +| 4 | 6.5 min | +47% | + +Two things to take from that. Wall clock has a floor of `V + M` no matter how far +you shard, so the returns fall off quickly — past four shards you pay a lot for +seconds. And sharding always costs *more* total compute than it saves in latency, +because every shard repeats `V`. If you are billed for runner minutes, or your +runner concurrency is contended, prefer the smallest `N` that gets you under your +target. + +On a short suite sharding is simply slower: this project's own 71-test suite goes +from 25s in one job to 41s across two plus a merge. + +## Using the GitHub Action + +The bundled action takes a `shard` input and handles the artifact upload, which +is the part that is easiest to get wrong: + +```yaml +jobs: + test: + strategy: + fail-fast: false + matrix: + shard: [1, 2] + steps: + # ...checkout, npm ci, dev server... + - uses: BRIKEV/twd-cli/.github/actions/run@main + with: + shard: ${{ matrix.shard }}/2 + + merge: + needs: [test] + if: ${{ !cancelled() }} + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: 24 + - run: npm ci + - uses: actions/download-artifact@v4 + with: + pattern: twd-run-* + path: .twd/shards + - run: npx twd-cli merge .twd/shards +``` + +The action uploads each shard's report as `twd-run-` with `if: always()`, +which is the layout `download-artifact` + `merge` expect. It also skips the +contract PR comment when `shard` is set, since a sharded run writes no contract +markdown per shard — post it from the merge job instead. + +## Caveats + +- **Coverage.** Each shard writes its own `coverage.json`; `merge` combines them + into `.nyc_output/out.json` — but only when the whole run is green, matching how + a single run behaves. `merge` reports how many shards contributed. +- **Missing shards are an error.** If a shard job dies before uploading, `merge` + refuses and names the gap rather than silently reporting 3 of 4 shards as a + complete green run. +- **Tests must register identically in every job.** Each shard fingerprints the + ordered list of `"suite > test"` paths it discovered and `merge` verifies they + match. Registering tests conditionally — behind a feature flag, a date, + `Math.random()` — makes the fingerprints diverge and `merge` will say so. + (Paths rather than internal test ids: `twd-js` assigns those at registration + time and they differ on every page load, so each shard's browser sees its own.) +- **`maxFailures` is per shard.** Four shards at the default of 10 can reach 40 + failures between them before all four bail. +- **`--test` and `--shard` compose:** filters resolve first, then the filtered list + is sharded. As with any filtered run, coverage is skipped. +- **Recording** produces one clip per shard; they are not concatenated. +- **A missing shard leaves no merged report on disk.** `merge` throws before it + writes `.twd/merged-run.json`, so a CI step that uploads that path with + `if: always()` will find nothing when a shard is missing. The error message on + stderr is the diagnosis in that case. +- **`record.filename` collides under sharding.** Only the *derived* recording + filename is per-shard. If `record.filename` is set explicitly in + `twd.config.json`, every shard writes to the same video path. Use the derived + name, or a per-shard `--record-dir`, when recording a sharded run. From e508207abec385227681d26844237d8ac5560a0a Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Tue, 25 Aug 2026 11:00:42 +0200 Subject: [PATCH 25/26] docs: mark sharding as beta and give it a runnable action example MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Sharding ships beta on purpose. It is strictly additive — a run without --shard writes the same files, prints the same output and exits the same way as 1.4.0 — so enabling it cannot disturb an existing pipeline. What is not yet a stable contract is which tests land in which shard: today each shard takes every nth test, and grouping by top-level describe so a suite always stays in one shard is the likely direction. Saying that up front makes the later change a documented evolution instead of a surprise. The beta status is now stated in the README section, the table of contents, the CHANGELOG entry, and `twd-cli` help, so it is visible wherever someone meets the flag rather than only in the docs. docs/sharding.md becomes a standalone guide: a complete copy-pasteable workflow using the bundled action rather than one abbreviated with elisions, a raw-CLI variant for people not using the action, the contract-report snippet for the merge job, and the three load-bearing conditions as a table naming what breaks without each. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 1 + README.md | 6 +- bin/twd-cli.js | 13 ++-- docs/sharding.md | 168 +++++++++++++++++++++++++++++++---------------- 4 files changed, 125 insertions(+), 63 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index cac57a1..4632a3b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,7 @@ * feat(merge): `npx twd-cli merge ` joins shard reports into one report covering test results, coverage and contract validation, prints a single summary with a per-shard breakdown, and owns the exit code * feat(merge): a missing shard report is an error naming the gap, not a silently incomplete report. Shards also fingerprint the test list they discovered, so shards that saw different test sets refuse to merge * chore(packaging): a `files` allowlist in package.json — the published package is now just `bin/`, `src/`, `README.md`, `CHANGELOG.md` and `LICENSE`. `tests/`, `test-example-app/`, `docs/` and the repo tooling were all being published and no longer are, taking the tarball from ~209 kB to ~33 kB (99 files to 25, ~850 kB to ~101 kB unpacked). Nothing that was importable before has moved +* note: **sharding ships as a beta feature.** It is strictly additive, so a run without `--shard` is unaffected, but which tests land in which shard is not yet a stable contract — a later release is likely to group by top-level `describe` so a suite always stays in one shard * note: no behavior change without `--shard`. A plain run writes the same files, prints the same output, and exits the same way as 1.4.0 Sharding needs three things right in the workflow: `fail-fast: false` on the diff --git a/README.md b/README.md index 6e4e0e6..bea5e1e 100644 --- a/README.md +++ b/README.md @@ -7,7 +7,7 @@ CI/CD runner for [TWD (Test while developing)](https://brikev.github.io/twd/) - [Recording](#recording): capture a run to video, paced so it is watchable - [Contract Validation](#contract-validation): check your mocks against OpenAPI specs - [CI/CD Integration](#cicd-integration): GitHub Action and custom setups -- [Sharding across CI jobs](#sharding-across-ci-jobs): split a long run across parallel jobs ([details](docs/sharding.md)) +- [Sharding across CI jobs](#sharding-across-ci-jobs) **(beta)**: split a long run across parallel jobs ([details](docs/sharding.md)) - [How It Works](#how-it-works) - [Requirements](#requirements) @@ -352,6 +352,10 @@ Failed validations are included in a collapsible details section with a link to ## Sharding across CI jobs +> **Beta.** Strictly additive: a run without `--shard` behaves exactly as before, +> so turning this on cannot affect your existing pipeline. How tests are assigned +> to shards may still change — see [docs/sharding.md](docs/sharding.md). + Long suites can be split across parallel CI jobs. Each shard runs one slice of the suite and writes a report; `twd-cli merge` joins them into a single summary and owns the exit code. diff --git a/bin/twd-cli.js b/bin/twd-cli.js index 62c097d..5c6a15b 100755 --- a/bin/twd-cli.js +++ b/bin/twd-cli.js @@ -47,10 +47,10 @@ Usage: contains (case-insensitive). Repeatable; multiple --test values are OR'd. npx twd-cli run --record Record the run to a video file - npx twd-cli run --shard 2/4 Run only this shard's slice of the suite - and write a report to ./.twd/run - npx twd-cli merge Merge shard reports from into one - report, and exit 1 if the whole run failed + npx twd-cli run --shard 2/4 (beta) Run only this shard's slice of the + suite and write a report to ./.twd/run + npx twd-cli merge (beta) Merge shard reports from into + one report, exit 1 if the run failed Examples: npx twd-cli run --test "shows error" @@ -60,9 +60,10 @@ Examples: Options: --test "" Filter tests by "suite > test" path (repeatable, OR'd) - --shard / Run slice i of n. Each shard discovers the whole - suite and takes every nth test, so the test count + --shard / (beta) Run slice i of n. Each shard discovers the + whole suite and takes every nth test, so the count never has to be known in advance. Implies a report. + Which tests land in which shard may change. --report-dir Where to write the shard report (default ./.twd/run) --out merge only: where to write the merged report (default ./.twd/merged-run.json) diff --git a/docs/sharding.md b/docs/sharding.md index 17dbe80..0b869a1 100644 --- a/docs/sharding.md +++ b/docs/sharding.md @@ -1,11 +1,21 @@ # Sharding across CI jobs -A single run walks the whole suite in one browser. `--shard` splits it across -parallel CI jobs instead. +> **Beta.** Sharding is new and marked beta on purpose. It is strictly additive — +> a run without `--shard` behaves exactly as it did before, writes the same +> files, and exits the same way — so enabling it cannot affect your existing +> pipeline. What may still change is **how tests are assigned to shards**: today +> each shard takes every nth test from the discovered list, and a future release +> is likely to group by top-level `describe` instead, so a suite always stays in +> one shard. Do not build anything that depends on *which* tests land in a given +> shard. Everything else — the flags, the report files, `merge`'s output and exit +> code — is stable. + +A single run walks the whole suite in one browser. Sharding splits it across +parallel CI jobs instead, then joins the results back into one report. ```bash npx twd-cli run --shard 2/4 # "I am job 2 of 4" -npx twd-cli merge .twd/shards # join the reports back together +npx twd-cli merge .twd/shards # join the reports, decide the exit code ``` The `4` is how many jobs you are running, **not** how many tests exist. You never @@ -14,47 +24,130 @@ suite exactly as a normal run does, and keeps every 4th test. Add tests and the same 4 jobs just split more of them. Each shard writes `run.json` and `coverage.json` to `./.twd/run` (change it with -`--report-dir`). `merge` reads the downloaded shard directories, combines the test +`--report-dir`). `merge` reads the downloaded shard directories, combines test results, coverage and contract validation, prints one summary, and exits non-zero if anything failed anywhere. +## A complete workflow + +This runs as-is. The bundled action installs Chrome, runs the shard, and uploads +its report under the name `merge` expects. + ```yaml +name: TWD tests (sharded) + +on: + pull_request: + branches: [main] + jobs: test: + runs-on: ubuntu-latest strategy: - fail-fast: false # or one red shard cancels the rest + # Without this, the first red shard cancels its siblings and the merge job + # sees gaps it cannot tell apart from a shard that crashed. + fail-fast: false matrix: shard: [1, 2, 3, 4] + steps: - # ...checkout, npm ci, chrome, dev server... - - run: npx twd-cli run --shard ${{ matrix.shard }}/4 - - uses: actions/upload-artifact@v4 - if: always() # a red shard must still upload + - uses: actions/checkout@v5 + + - uses: actions/setup-node@v5 with: - name: twd-run-${{ matrix.shard }} - path: .twd/run + node-version: 24 + cache: npm + + - run: npm ci + + - name: Start the dev server + run: | + nohup npm run dev > dev.log 2>&1 & + npx wait-on http://localhost:5173 --timeout 60000 + + - name: Run this shard + uses: BRIKEV/twd-cli/.github/actions/run@main + with: + shard: ${{ matrix.shard }}/4 merge: + runs-on: ubuntu-latest needs: [test] - if: ${{ !cancelled() }} # runs even though a shard went red + # Runs even though a shard job may have exited 1. Without this a red shard + # short-circuits the workflow and the merged summary never prints. + if: ${{ !cancelled() }} + steps: - uses: actions/checkout@v5 + - uses: actions/setup-node@v5 with: node-version: 24 cache: npm + - run: npm ci + - uses: actions/download-artifact@v4 with: pattern: twd-run-* path: .twd/shards - - run: npx twd-cli merge .twd/shards + + - name: Merge the shard reports + run: npx twd-cli merge .twd/shards +``` + +`merge` owns the final exit code: it fails if any test failed in any shard, if a +contract was violated in `error` mode, or if a shard report is missing entirely. + +### Posting the contract report + +A sharded run deliberately writes no contract markdown per shard — each would +overwrite the others with a fraction of the mocks — so `merge` writes it, and the +PR comment belongs in the merge job: + +```yaml + merge: + permissions: + contents: read + pull-requests: write + steps: + # ...as above, through "Merge the shard reports"... + + - name: Post contract report to PR + if: github.event_name == 'pull_request' && hashFiles('.twd/contract-report.md') != '' + env: + GH_TOKEN: ${{ github.token }} + run: gh pr comment "${{ github.event.pull_request.number }}" --body-file .twd/contract-report.md ``` -Those three conditions are easy to miss and each one breaks the run: -`fail-fast: false` stops a red shard cancelling its siblings, `if: always()` on -upload keeps a red shard's report, and `if: ${{ !cancelled() }}` on merge lets the -summary print at all. +## Without the bundled action + +If you drive the CLI directly, you own the two steps the action was doing for +you — installing Chrome, and uploading the report with `if: always()`: + +```yaml + - run: npx puppeteer browsers install chrome + - run: npx twd-cli run --shard ${{ matrix.shard }}/4 + - uses: actions/upload-artifact@v4 + # A red shard must still upload, or merge cannot tell "this shard failed" + # from "this shard never ran". + if: always() + with: + name: twd-run-${{ matrix.shard }} + path: .twd/run + if-no-files-found: error +``` + +## The three conditions that matter + +Each of these breaks a sharded run in a different way, and all three are easy to +leave out: + +| Condition | Where | What breaks without it | +|---|---|---| +| `fail-fast: false` | the shard matrix | the first red shard cancels its siblings, and `merge` reports their reports as missing | +| `if: always()` | the shard's artifact upload | a red shard uploads nothing, so `merge` cannot distinguish failure from a crash | +| `if: ${{ !cancelled() }}` | the merge job | a red shard short-circuits the workflow and the merged summary never prints | ## When sharding pays @@ -86,45 +179,6 @@ target. On a short suite sharding is simply slower: this project's own 71-test suite goes from 25s in one job to 41s across two plus a merge. -## Using the GitHub Action - -The bundled action takes a `shard` input and handles the artifact upload, which -is the part that is easiest to get wrong: - -```yaml -jobs: - test: - strategy: - fail-fast: false - matrix: - shard: [1, 2] - steps: - # ...checkout, npm ci, dev server... - - uses: BRIKEV/twd-cli/.github/actions/run@main - with: - shard: ${{ matrix.shard }}/2 - - merge: - needs: [test] - if: ${{ !cancelled() }} - steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 - with: - node-version: 24 - - run: npm ci - - uses: actions/download-artifact@v4 - with: - pattern: twd-run-* - path: .twd/shards - - run: npx twd-cli merge .twd/shards -``` - -The action uploads each shard's report as `twd-run-` with `if: always()`, -which is the layout `download-artifact` + `merge` expect. It also skips the -contract PR comment when `shard` is set, since a sharded run writes no contract -markdown per shard — post it from the merge job instead. - ## Caveats - **Coverage.** Each shard writes its own `coverage.json`; `merge` combines them @@ -152,3 +206,5 @@ markdown per shard — post it from the merge job instead. filename is per-shard. If `record.filename` is set explicitly in `twd.config.json`, every shard writes to the same video path. Use the derived name, or a per-shard `--record-dir`, when recording a sharded run. +- **Assignment may change.** See the beta note at the top: which tests land in + which shard is not part of the stable contract yet. From a5b75705f15d0254d4db66c4a58caf6ee2216053 Mon Sep 17 00:00:00 2001 From: kevinccbsg Date: Tue, 25 Aug 2026 22:20:37 +0200 Subject: [PATCH 26/26] chore(release): 1.5.0 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Releases as a normal version rather than a prerelease. The sharding *feature* is documented as beta — in the README, the docs, the CHANGELOG entry and `twd-cli` help — but the release itself is stable, so `npm install twd-cli` picks it up and nobody has to opt in through a dist-tag to get the rest of the version. Both package-lock version fields moved with it, regenerated through lock:linux. Also drops the implementation plan from the repo; the design spec is the document worth keeping. Co-Authored-By: Claude Opus 5 (1M context) --- CHANGELOG.md | 10 +- .../2026-08-19-shardable-run-artifacts.md | 2858 ----------------- package-lock.json | 4 +- package.json | 2 +- 4 files changed, 8 insertions(+), 2866 deletions(-) delete mode 100644 docs/superpowers/plans/2026-08-19-shardable-run-artifacts.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 4632a3b..a146799 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,4 +1,4 @@ -## 1.5.0-beta.0 (2026-08-19) +## 1.5.0 (2026-08-25) * feat(shard): `--shard /` runs one slice of the suite so a run can be split across parallel CI jobs. Each shard discovers the whole suite itself and takes every nth test, so the test count never has to be known in advance * feat(shard): a sharded run writes `run.json` and `coverage.json` to `./.twd/run` (`--report-dir` to change it) — the first machine-readable output twd-cli has had @@ -10,11 +10,11 @@ Sharding needs three things right in the workflow: `fail-fast: false` on the matrix, `if: always()` on the shard's artifact upload, and -`if: ${{ !cancelled() }}` on the merge job. See "Sharding across CI jobs" in the -README. +`if: ${{ !cancelled() }}` on the merge job. Each one breaks the run differently +if left out. See [docs/sharding.md](docs/sharding.md) for a runnable workflow. -This is a prerelease, published under the `beta` dist-tag: -`npm install twd-cli@beta`. +A normal release: `npm install twd-cli` gets it. The *sharding feature* is the +part marked beta — everything else in this version is stable. ## 1.4.0 (2026-07-28) diff --git a/docs/superpowers/plans/2026-08-19-shardable-run-artifacts.md b/docs/superpowers/plans/2026-08-19-shardable-run-artifacts.md deleted file mode 100644 index 2f3597f..0000000 --- a/docs/superpowers/plans/2026-08-19-shardable-run-artifacts.md +++ /dev/null @@ -1,2858 +0,0 @@ -# Shardable Run Artifacts Implementation Plan - -> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. - -**Goal:** Let a twd-cli run be split across parallel CI jobs with `--shard i/n`, each writing a machine-readable report plus raw coverage, and add `twd-cli merge ` to join them into one report covering tests, coverage, and contract validation. - -**Architecture:** Each shard boots its own browser, enumerates the whole suite as runs already do, and keeps every Nth test id (round-robin). It writes `run.json` + `coverage.json` to a report dir that CI uploads as an artifact. `merge` reads the downloaded shard dirs, validates they agree, concatenates them into a report of the **same shape**, and owns the exit code. Merged-equals-single shape makes merge associative and lets existing formatters render both. - -**Tech Stack:** Node ESM, vitest, Puppeteer (already present), `istanbul-lib-coverage` (new runtime dependency). - -**Spec:** `docs/superpowers/specs/2026-08-19-shardable-run-artifacts-design.md` - -## Global Constraints - -- **Strictly additive.** A run without `--shard` must behave exactly as 1.4.0 does: same console output, same files written, same exit code. Every behavior change is gated on `sharded` being true. -- Work happens on branch `feat/shardable-run-artifacts` (already checked out). Never commit to `main`. -- ESM only. `import`, no `require`. -- No test may require a real browser or a real ffmpeg binary. `node:child_process` and `page.screencast` stay mocked. -- `vi.mock('fs')` auto-mocks `fs.statSync` to return `undefined`. Anything reading a `Stats` must tolerate that. -- One `src/` module per responsibility, one `tests/.test.js` per module — the existing repo convention. -- After any dependency change run `npm run lock:linux` (Docker must be running). macOS npm never installs the wasm32-wasi optional packages, so it leaves their `@emnapi/*` transitive deps stale in the lock and `npm ci` breaks on Linux CI. -- Report schema version is `1`. Default report dir is `./.twd/run`. Default merged output is `./.twd/merged-run.json`. -- Final version is `1.5.0-beta.0`, published under the `beta` dist-tag. - -## File Structure - -**Create:** - -| File | Responsibility | -|---|---| -| `src/shard.js` | Parse `/`; round-robin id slicing | -| `src/runReport.js` | Build the report object; fingerprint the discovered test list. Pure, no I/O | -| `src/reportFiles.js` | Write a shard's report + coverage; discover and read shard dirs | -| `src/mergeCoverage.js` | Merge Istanbul coverage objects | -| `src/mergeReports.js` | Associative structural merge + consistency validation + derived totals | -| `src/mergeCommand.js` | Orchestrate `merge`: read, merge, write, render, decide exit code | -| `tests/shard.test.js`, `tests/runReport.test.js`, `tests/reportFiles.test.js`, `tests/mergeCoverage.test.js`, `tests/mergeReports.test.js`, `tests/mergeCommand.test.js` | One per module | - -**Modify:** - -| File | Change | -|---|---| -| `src/parseArgs.js` | `--shard`, `--report-dir`; new `parseMergeArgs` | -| `src/index.js` | Slice ids; build and write the report; gate the two behavior changes on `sharded` | -| `src/testSummary.js` | Optional `shards` / `computeMs` params for the merged breakdown | -| `bin/twd-cli.js` | `merge` subcommand + help text | -| `tests/parseArgs.test.js` | 8 full-object assertions gain the new keys | -| `tests/runTests.test.js` | Shard slicing, report writing, and the two non-regression assertions | -| `tests/testSummary.test.js` | Breakdown line rendering | -| `package.json` | `istanbul-lib-coverage` dependency; version `1.5.0-beta.0` | -| `.github/workflows/e2e.yml` | A 2-shard + merge job | -| `README.md`, `CHANGELOG.md` | Document the flags and the command | - -**Why `mergeCommand.js` is separate from `mergeReports.js`:** `mergeReports` must stay pure and associative to be property-testable. Completeness checking (is `1..total` fully covered?) cannot live there, because a partial merge of 2 of 3 shards is a legal intermediate value — enforcing completeness inside the merge would make `merge(merge(a,b),c)` throw. So `mergeReports` validates *consistency* (things preserved under partial merge) and `mergeCommand` enforces *completeness*. - ---- - -### Task 1: Shard spec parsing and id slicing - -**Files:** -- Create: `src/shard.js` -- Test: `tests/shard.test.js` - -**Interfaces:** -- Consumes: nothing. -- Produces: `parseShardSpec(value) -> { index: number, total: number }` (throws `Error` on invalid input); `selectShardIds(ids: string[], index: number, total: number) -> string[]`. - -- [ ] **Step 1: Write the failing test** - -Create `tests/shard.test.js`: - -```js -import { describe, it, expect } from 'vitest'; -import { parseShardSpec, selectShardIds } from '../src/shard.js'; - -describe('parseShardSpec', () => { - it('parses /', () => { - expect(parseShardSpec('2/4')).toEqual({ index: 2, total: 4 }); - expect(parseShardSpec('1/1')).toEqual({ index: 1, total: 1 }); - expect(parseShardSpec(' 3/4 ')).toEqual({ index: 3, total: 4 }); - }); - - // Unlike --record-speed, a bad --shard must never be silently ignored: it - // would run zero tests and exit 0, reading as a green build that tested - // nothing. - it('throws when the index exceeds the total', () => { - expect(() => parseShardSpec('5/4')).toThrow(/between 1 and 4/); - }); - - it('throws on a zero or negative index', () => { - expect(() => parseShardSpec('0/4')).toThrow(/between 1 and 4/); - expect(() => parseShardSpec('-1/4')).toThrow(/Expected \//); - }); - - it('throws on a zero total', () => { - expect(() => parseShardSpec('2/0')).toThrow(/at least 1/); - }); - - it('throws on unparseable input', () => { - for (const bad of ['abc', '', '2', '2/', '/4', '2.5/4', '2/4/6', undefined, null]) { - expect(() => parseShardSpec(bad)).toThrow(/Expected \//); - } - }); - - it('names the offending value in the message', () => { - expect(() => parseShardSpec('9/2')).toThrow(/"9\/2"/); - }); -}); - -describe('selectShardIds', () => { - const ids = Array.from({ length: 12 }, (_, i) => `t${i}`); - - it('takes every Nth id at its own offset', () => { - expect(selectShardIds(ids, 1, 4)).toEqual(['t0', 't4', 't8']); - expect(selectShardIds(ids, 2, 4)).toEqual(['t1', 't5', 't9']); - expect(selectShardIds(ids, 4, 4)).toEqual(['t3', 't7', 't11']); - }); - - it('returns everything when total is 1', () => { - expect(selectShardIds(ids, 1, 1)).toEqual(ids); - }); - - // 3 tests across 4 shards leaves the fourth with nothing. Legal, not an error. - it('returns an empty slice when there are fewer ids than shards', () => { - expect(selectShardIds(['a', 'b', 'c'], 4, 4)).toEqual([]); - expect(selectShardIds([], 1, 4)).toEqual([]); - }); - - // The property that makes sharding trustworthy: nothing lost, nothing doubled. - it('partitions the input — every id lands in exactly one shard', () => { - const many = Array.from({ length: 37 }, (_, i) => `t${i}`); - const total = 5; - const slices = Array.from({ length: total }, (_, i) => selectShardIds(many, i + 1, total)); - const counts = new Map(); - for (const id of slices.flat()) counts.set(id, (counts.get(id) ?? 0) + 1); - expect(counts.size).toBe(many.length); - expect([...counts.values()]).toEqual(many.map(() => 1)); - }); - - it('does not mutate its input', () => { - const input = ['a', 'b', 'c']; - selectShardIds(input, 1, 2); - expect(input).toEqual(['a', 'b', 'c']); - }); -}); -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `npx vitest --run tests/shard.test.js` -Expected: FAIL — `Failed to load ../src/shard.js`. - -- [ ] **Step 3: Write the implementation** - -Create `src/shard.js`: - -```js -// Parses a "/" shard spec, e.g. "2/4" for job 2 of 4. -// -// This throws where src/parseArgs.js silently ignores a malformed -// --record-speed, and the divergence is deliberate: "--shard 5/4" would select -// no tests and exit 0, which reads as a green build that tested nothing. -export function parseShardSpec(value) { - const raw = typeof value === 'string' ? value.trim() : value; - const match = /^(\d+)\/(\d+)$/.exec(String(raw ?? '')); - if (!match) { - throw new Error( - `Invalid --shard "${value}". Expected /, e.g. --shard 2/4.` - ); - } - - const index = Number(match[1]); - const total = Number(match[2]); - - if (total < 1) { - throw new Error(`Invalid --shard "${value}". Total must be at least 1.`); - } - if (index < 1 || index > total) { - throw new Error( - `Invalid --shard "${value}". Index must be between 1 and ${total}.` - ); - } - - return { index, total }; -} - -// Round-robin slice of an ordered id list. `index` is 1-based. -// -// Round-robin rather than contiguous: it balances better when adjacent tests -// have similar cost, and nothing in twd-js requires a suite to run -// contiguously, so locality buys nothing. An empty result is a legal outcome — -// 3 tests across 4 shards leaves the fourth with nothing to run. -export function selectShardIds(ids, index, total) { - return ids.filter((_, i) => i % total === index - 1); -} -``` - -- [ ] **Step 4: Run the test to verify it passes** - -Run: `npx vitest --run tests/shard.test.js` -Expected: PASS, 9 tests. - -- [ ] **Step 5: Commit** - -```bash -git add src/shard.js tests/shard.test.js -git commit -m "feat(shard): parse shard specs and slice test ids round-robin" -``` - ---- - -### Task 2: `--shard` and `--report-dir` flags, plus `parseMergeArgs` - -**Files:** -- Modify: `src/parseArgs.js` -- Modify: `tests/parseArgs.test.js` (8 existing assertions at lines 6, 10, 17, 24, 31, 35, 71, 93) -- Test: `tests/parseArgs.test.js` - -**Interfaces:** -- Consumes: `parseShardSpec` from `src/shard.js` (Task 1). -- Produces: `parseRunArgs(argv) -> { testFilters: string[], record: object, shard: {index,total}|null, reportDir: string|null }`; `parseMergeArgs(argv) -> { dir: string|null, out: string|null }`. - -`parseRunArgs` now always returns `shard` and `reportDir`, defaulting to `null`. That is why the 8 existing full-object assertions must be updated — they use `toEqual` on the whole return value. - -- [ ] **Step 1: Update the 8 existing assertions** - -In `tests/parseArgs.test.js`, add `shard: null, reportDir: null` to every full-object `toEqual`. The two single-line ones become: - -```js -// line 6 -expect(parseRunArgs([])).toEqual({ testFilters: [], record: {}, shard: null, reportDir: null }); -// line 31 -expect(parseRunArgs(['--test'])).toEqual({ testFilters: [], record: {}, shard: null, reportDir: null }); -``` - -The six multi-line ones (lines 10, 17, 24, 35, 71, 93) each gain two properties, e.g.: - -```js -expect(parseRunArgs(['--test', 'shows error'])).toEqual({ - testFilters: ['shows error'], - record: {}, - shard: null, - reportDir: null, -}); -``` - -- [ ] **Step 2: Write the failing tests for the new flags** - -Append to `tests/parseArgs.test.js`: - -```js -describe('parseRunArgs shard and report flags', () => { - it('parses --shard in both forms', () => { - expect(parseRunArgs(['--shard', '2/4']).shard).toEqual({ index: 2, total: 4 }); - expect(parseRunArgs(['--shard=3/4']).shard).toEqual({ index: 3, total: 4 }); - }); - - it('throws on an invalid --shard instead of ignoring it', () => { - expect(() => parseRunArgs(['--shard', '5/4'])).toThrow(/Invalid --shard/); - expect(() => parseRunArgs(['--shard=abc'])).toThrow(/Invalid --shard/); - }); - - it('throws on a trailing --shard with no value', () => { - expect(() => parseRunArgs(['--shard'])).toThrow(/Invalid --shard/); - }); - - it('parses --report-dir in both forms', () => { - expect(parseRunArgs(['--report-dir', './out']).reportDir).toBe('./out'); - expect(parseRunArgs(['--report-dir=./out']).reportDir).toBe('./out'); - }); - - it('ignores a trailing --report-dir with no value', () => { - expect(parseRunArgs(['--report-dir']).reportDir).toBeNull(); - }); - - it('combines --shard with --test filters and record flags', () => { - expect(parseRunArgs(['--shard', '2/4', '--test', 'Login', '--record'])).toEqual({ - testFilters: ['Login'], - record: { enabled: true }, - shard: { index: 2, total: 4 }, - reportDir: null, - }); - }); -}); - -describe('parseMergeArgs', () => { - it('reads the directory as the first positional', () => { - expect(parseMergeArgs(['.twd/shards'])).toEqual({ dir: '.twd/shards', out: null }); - }); - - it('parses --out in both forms', () => { - expect(parseMergeArgs(['.twd/shards', '--out', 'merged.json'])) - .toEqual({ dir: '.twd/shards', out: 'merged.json' }); - expect(parseMergeArgs(['.twd/shards', '--out=merged.json'])) - .toEqual({ dir: '.twd/shards', out: 'merged.json' }); - }); - - it('returns a null dir when none is given', () => { - expect(parseMergeArgs([])).toEqual({ dir: null, out: null }); - expect(parseMergeArgs(['--out=merged.json'])).toEqual({ dir: null, out: 'merged.json' }); - }); - - it('takes only the first positional as the directory', () => { - expect(parseMergeArgs(['a', 'b']).dir).toBe('a'); - }); -}); -``` - -Update the import at the top of the file: - -```js -import { parseRunArgs, parseMergeArgs } from "../src/parseArgs.js"; -``` - -- [ ] **Step 3: Run the tests to verify they fail** - -Run: `npx vitest --run tests/parseArgs.test.js` -Expected: FAIL — `parseMergeArgs is not a function`, and the new shard assertions fail on `undefined`. - -- [ ] **Step 4: Implement the flags** - -In `src/parseArgs.js`, add the import at the top: - -```js -import { parseShardSpec } from './shard.js'; -``` - -Hoist the existing `readValue` closure to module scope so `parseMergeArgs` can -share it rather than duplicating it. Delete the `const readValue = ...` block -from inside `parseRunArgs` and add above it: - -```js -// Reads a flag's value in either `--flag value` or `--flag=value` form, and -// reports how many tokens it consumed. Shared by both parsers. -function readValue(argv, token, prefix, index) { - if (token === prefix) { - return { value: argv[index + 1], consumed: argv[index + 1] !== undefined ? 2 : 1 }; - } - return { value: token.slice(prefix.length + 1), consumed: 1 }; -} -``` - -Then update every call site inside `parseRunArgs` to pass `argv` first — there -are four existing ones (`--test`, `--record-dir`, `--record-speed`, -`--record-pace`), each becoming e.g. `readValue(argv, token, '--test', i)`. The -17 existing tests in `tests/parseArgs.test.js` guard this refactor. - -Inside `parseRunArgs`, add two declarations next to the existing ones: - -```js -export function parseRunArgs(argv) { - const testFilters = []; - const record = {}; - let shard = null; - let reportDir = null; -``` - -Add two branches to the token loop, after the `--test` branch: - -```js - } else if (token === '--shard' || token.startsWith('--shard=')) { - const { value, consumed } = readValue(argv, token, '--shard', i); - // Throws on a malformed spec. A silently-ignored --shard would run zero - // tests and exit 0. - shard = parseShardSpec(value); - i += consumed - 1; - } else if (token === '--report-dir' || token.startsWith('--report-dir=')) { - const { value, consumed } = readValue(argv, token, '--report-dir', i); - if (value !== undefined) reportDir = value; - i += consumed - 1; -``` - -Change the return statement: - -```js - return { testFilters, record, shard, reportDir }; -} -``` - -Append `parseMergeArgs` to the same file: - -```js -// `twd-cli merge [--out ]`. The directory is the first positional -// token; anything after the first is ignored. -export function parseMergeArgs(argv) { - let dir = null; - let out = null; - - for (let i = 0; i < argv.length; i++) { - const token = argv[i]; - - if (token === '--out' || token.startsWith('--out=')) { - const { value, consumed } = readValue(argv, token, '--out', i); - if (value !== undefined) out = value; - i += consumed - 1; - } else if (!token.startsWith('--') && dir === null) { - dir = token; - } - } - - return { dir, out }; -} -``` - -- [ ] **Step 5: Run the tests to verify they pass** - -Run: `npx vitest --run tests/parseArgs.test.js` -Expected: PASS — the 17 original tests plus 10 new ones. - -- [ ] **Step 6: Commit** - -```bash -git add src/parseArgs.js tests/parseArgs.test.js -git commit -m "feat(cli): add --shard, --report-dir, and merge arg parsing" -``` - ---- - -### Task 3: Build the run report and fingerprint discovery - -**Files:** -- Create: `src/runReport.js` -- Test: `tests/runReport.test.js` - -**Interfaces:** -- Consumes: nothing. -- Produces: `REPORT_SCHEMA_VERSION` (number, `1`); `fingerprintTests(orderedIds: string[], filters: string[]) -> string`; `buildRunReport(options) -> report`. `buildRunReport` takes `{ shard, startedAt, endedAt, allTestIds, filters, handlers, tests, executed, notRun, stoppedEarly, coverageFile, recording, contracts }` where `startedAt`/`endedAt` are epoch milliseconds and `shard` is `{index,total}`. - -- [ ] **Step 1: Write the failing test** - -Create `tests/runReport.test.js`: - -```js -import { describe, it, expect } from 'vitest'; -import { buildRunReport, fingerprintTests, REPORT_SCHEMA_VERSION } from '../src/runReport.js'; - -const handlers = [ - { id: 's1', name: 'Login', parent: null, type: 'suite' }, - { id: 't1', name: 'works', parent: 's1', type: 'test' }, -]; - -function build(overrides = {}) { - return buildRunReport({ - shard: { index: 2, total: 4 }, - startedAt: 1_000, - endedAt: 4_500, - allTestIds: ['t1', 't2', 't3'], - filters: [], - handlers, - tests: [{ id: 't1', status: 'pass' }], - executed: 1, - notRun: 0, - stoppedEarly: false, - ...overrides, - }); -} - -describe('fingerprintTests', () => { - it('is stable for the same input', () => { - expect(fingerprintTests(['a', 'b'])).toBe(fingerprintTests(['a', 'b'])); - }); - - it('is prefixed with the algorithm', () => { - expect(fingerprintTests(['a'])).toMatch(/^sha256:[0-9a-f]{64}$/); - }); - - // Order matters: round-robin slicing is only correct if every shard sees the - // same list in the same order. - it('changes when the id order changes', () => { - expect(fingerprintTests(['a', 'b'])).not.toBe(fingerprintTests(['b', 'a'])); - }); - - it('changes when the id set changes', () => { - expect(fingerprintTests(['a', 'b'])).not.toBe(fingerprintTests(['a', 'b', 'c'])); - }); - - it('changes when the filters differ', () => { - expect(fingerprintTests(['a'], ['Login'])).not.toBe(fingerprintTests(['a'], ['Cart'])); - expect(fingerprintTests(['a'], [])).not.toBe(fingerprintTests(['a'], ['Login'])); - }); - - // Filters are OR'd, so their order is not meaningful and must not split - // otherwise-identical shards. - it('ignores the order the filters were given in', () => { - expect(fingerprintTests(['a'], ['Login', 'Cart'])) - .toBe(fingerprintTests(['a'], ['Cart', 'Login'])); - }); -}); - -describe('buildRunReport', () => { - it('stamps the schema version', () => { - expect(build().schemaVersion).toBe(REPORT_SCHEMA_VERSION); - }); - - it('wraps a single shard descriptor in an array', () => { - const report = build(); - expect(report.shards).toHaveLength(1); - expect(report.shards[0]).toMatchObject({ - index: 2, total: 4, executed: 1, notRun: 0, failed: 0, stoppedEarly: false, - }); - }); - - it('derives durationMs and ISO timestamps from epoch millis', () => { - const shard = build().shards[0]; - expect(shard.durationMs).toBe(3500); - expect(shard.startedAt).toBe(new Date(1_000).toISOString()); - expect(shard.endedAt).toBe(new Date(4_500).toISOString()); - }); - - it('counts this shard\'s failures', () => { - const report = build({ - tests: [ - { id: 't1', status: 'pass' }, - { id: 't2', status: 'fail', error: 'boom' }, - { id: 't3', status: 'skip' }, - ], - }); - expect(report.shards[0].failed).toBe(1); - }); - - it('records total discovered tests and the fingerprint', () => { - const report = build(); - expect(report.discovery.totalTests).toBe(3); - expect(report.discovery.fingerprint).toBe(fingerprintTests(['t1', 't2', 't3'], [])); - }); - - it('carries handlers and tests through untouched', () => { - const report = build(); - expect(report.handlers).toEqual(handlers); - expect(report.tests).toEqual([{ id: 't1', status: 'pass' }]); - }); - - it('copies the filters rather than aliasing them', () => { - const filters = ['Login']; - const report = build({ filters }); - filters.push('Cart'); - expect(report.selection.filters).toEqual(['Login']); - }); - - it('defaults contracts to an unconfigured empty block', () => { - expect(build().contracts).toEqual({ - configured: false, partial: false, results: [], skipped: [], - }); - }); - - it('passes a contracts block through when given', () => { - const contracts = { configured: true, partial: true, results: [{ alias: 'a' }], skipped: [] }; - expect(build({ contracts }).contracts).toEqual(contracts); - }); - - it('defaults coverageFile and recording to null', () => { - const shard = build().shards[0]; - expect(shard.coverageFile).toBeNull(); - expect(shard.recording).toBeNull(); - }); -}); -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `npx vitest --run tests/runReport.test.js` -Expected: FAIL — `Failed to load ../src/runReport.js`. - -- [ ] **Step 3: Write the implementation** - -Create `src/runReport.js`: - -```js -import crypto from 'node:crypto'; - -export const REPORT_SCHEMA_VERSION = 1; - -/** - * Hash of the full ordered test id list plus any active --test filters. - * - * Round-robin sharding is correct only if every job enumerates an identical - * test set in an identical order. That breaks silently if the app registers - * tests conditionally — a feature flag, a date, Math.random — or if two shard - * jobs did not build the same code: tests quietly never run and the build stays - * green. Shards compare fingerprints at merge time so it becomes an error. - * - * Filters are OR'd, so their order carries no meaning and is normalized away. - */ -export function fingerprintTests(orderedIds, filters = []) { - const payload = JSON.stringify({ - orderedIds, - filters: [...filters].sort(), - }); - const digest = crypto.createHash('sha256').update(payload).digest('hex'); - return `sha256:${digest}`; -} - -/** - * Assembles the on-disk run report. Pure: no I/O, no clock reads. - * - * `shards` is an array even for a single run, because a merged report has the - * same shape as a single-shard one. That is what makes merging associative and - * lets one set of formatters render both. - * - * `startedAt` and `endedAt` are epoch milliseconds; the report stores ISO - * strings plus the derived duration. - */ -export function buildRunReport({ - shard, - startedAt, - endedAt, - allTestIds, - filters = [], - handlers, - tests, - executed, - notRun, - stoppedEarly, - coverageFile = null, - recording = null, - contracts = null, -}) { - return { - schemaVersion: REPORT_SCHEMA_VERSION, - shards: [ - { - index: shard.index, - total: shard.total, - startedAt: new Date(startedAt).toISOString(), - endedAt: new Date(endedAt).toISOString(), - durationMs: endedAt - startedAt, - executed, - notRun, - // Merged reports do not record which shard ran a test, so the per-shard - // breakdown line could not be rendered without this count. - failed: tests.filter((t) => t.status === 'fail').length, - stoppedEarly, - coverageFile, - recording, - }, - ], - discovery: { - totalTests: allTestIds.length, - fingerprint: fingerprintTests(allTestIds, filters), - }, - selection: { filters: [...filters] }, - handlers, - tests, - contracts: contracts ?? { - configured: false, - partial: false, - results: [], - skipped: [], - }, - }; -} -``` - -- [ ] **Step 4: Run the test to verify it passes** - -Run: `npx vitest --run tests/runReport.test.js` -Expected: PASS, 16 tests. - -- [ ] **Step 5: Commit** - -```bash -git add src/runReport.js tests/runReport.test.js -git commit -m "feat(report): build run reports and fingerprint discovered tests" -``` - ---- - -### Task 4: Read and write shard report files - -**Files:** -- Create: `src/reportFiles.js` -- Test: `tests/reportFiles.test.js` - -**Interfaces:** -- Consumes: nothing. -- Produces: `DEFAULT_REPORT_DIR` (`'./.twd/run'`), `DEFAULT_MERGED_OUT` (`'./.twd/merged-run.json'`), `RUN_REPORT_FILE` (`'run.json'`), `COVERAGE_FILE` (`'coverage.json'`); `writeRunReport(dir, report, coverage) -> { reportPath, coveragePath|null }`; `readShardReports(dir) -> Array<{ dir, report }>`; `readShardCoverage(shardDir, coverageFile) -> object|null`. - -- [ ] **Step 1: Write the failing test** - -Create `tests/reportFiles.test.js`: - -```js -import { describe, it, expect, vi, beforeEach } from 'vitest'; - -vi.mock('fs'); - -import fs from 'fs'; -import { - writeRunReport, - readShardReports, - readShardCoverage, - RUN_REPORT_FILE, - COVERAGE_FILE, - DEFAULT_REPORT_DIR, -} from '../src/reportFiles.js'; - -const report = { schemaVersion: 1, tests: [] }; - -describe('writeRunReport', () => { - beforeEach(() => { - vi.clearAllMocks(); - }); - - it('creates the directory recursively', () => { - writeRunReport('.twd/run', report, null); - expect(fs.mkdirSync).toHaveBeenCalledWith('.twd/run', { recursive: true }); - }); - - it('writes pretty-printed JSON so the report is readable by eye', () => { - writeRunReport('.twd/run', report, null); - const [file, body] = vi.mocked(fs.writeFileSync).mock.calls[0]; - expect(file).toBe(`.twd/run/${RUN_REPORT_FILE}`); - expect(body).toBe(`${JSON.stringify(report, null, 2)}\n`); - }); - - it('does not write a coverage file when there is no coverage', () => { - writeRunReport('.twd/run', report, null); - expect(fs.writeFileSync).toHaveBeenCalledTimes(1); - expect(writeRunReport('.twd/run', report, null).coveragePath).toBeNull(); - }); - - // Coverage stays raw and unformatted: it is machine input for nyc, routinely - // several megabytes, and pretty-printing it would double the artifact size. - it('writes coverage compactly alongside the report', () => { - const coverage = { '/a.js': { s: { 0: 1 } } }; - const result = writeRunReport('.twd/run', report, coverage); - expect(fs.writeFileSync).toHaveBeenCalledWith( - `.twd/run/${COVERAGE_FILE}`, - JSON.stringify(coverage), - ); - expect(result.coveragePath).toBe(`.twd/run/${COVERAGE_FILE}`); - }); -}); - -describe('readShardReports', () => { - beforeEach(() => { - vi.clearAllMocks(); - }); - - // This is download-artifact's layout: one directory per artifact name. - it('reads run.json from each child directory', () => { - vi.mocked(fs.existsSync).mockImplementation((p) => String(p).endsWith(RUN_REPORT_FILE)); - vi.mocked(fs.readdirSync).mockReturnValue([ - { name: 'twd-run-1', isDirectory: () => true }, - { name: 'twd-run-2', isDirectory: () => true }, - { name: 'notes.txt', isDirectory: () => false }, - ]); - vi.mocked(fs.readFileSync).mockReturnValue(JSON.stringify(report)); - - const found = readShardReports('.twd/shards'); - - expect(found.map((f) => f.dir)).toEqual([ - '.twd/shards', - '.twd/shards/twd-run-1', - '.twd/shards/twd-run-2', - ]); - expect(found[0].report).toEqual(report); - }); - - it('skips child directories with no run.json', () => { - vi.mocked(fs.existsSync).mockImplementation((p) => String(p).includes('twd-run-1')); - vi.mocked(fs.readdirSync).mockReturnValue([ - { name: 'twd-run-1', isDirectory: () => true }, - { name: 'empty', isDirectory: () => true }, - ]); - vi.mocked(fs.readFileSync).mockReturnValue(JSON.stringify(report)); - - expect(readShardReports('.twd/shards').map((f) => f.dir)) - .toEqual(['.twd/shards/twd-run-1']); - }); - - it('returns an empty array when the directory does not exist', () => { - vi.mocked(fs.existsSync).mockReturnValue(false); - vi.mocked(fs.readdirSync).mockImplementation(() => { - throw Object.assign(new Error('ENOENT'), { code: 'ENOENT' }); - }); - expect(readShardReports('.twd/nope')).toEqual([]); - }); - - it('explains which file failed to parse', () => { - vi.mocked(fs.existsSync).mockReturnValue(true); - vi.mocked(fs.readdirSync).mockReturnValue([]); - vi.mocked(fs.readFileSync).mockReturnValue('{ not json'); - expect(() => readShardReports('.twd/shards')).toThrow(/run\.json/); - }); -}); - -describe('readShardCoverage', () => { - beforeEach(() => { - vi.clearAllMocks(); - }); - - it('reads the named coverage file from the shard directory', () => { - vi.mocked(fs.existsSync).mockReturnValue(true); - vi.mocked(fs.readFileSync).mockReturnValue('{"/a.js":{}}'); - expect(readShardCoverage('.twd/shards/twd-run-1', COVERAGE_FILE)).toEqual({ '/a.js': {} }); - }); - - it('returns null when the shard recorded no coverage file', () => { - expect(readShardCoverage('.twd/shards/twd-run-1', null)).toBeNull(); - expect(fs.existsSync).not.toHaveBeenCalled(); - }); - - // A shard whose coverage file is absent simply does not contribute; merge - // reports the contributor count rather than failing. - it('returns null when the file is missing on disk', () => { - vi.mocked(fs.existsSync).mockReturnValue(false); - expect(readShardCoverage('.twd/shards/twd-run-1', COVERAGE_FILE)).toBeNull(); - }); -}); - -describe('defaults', () => { - it('defaults the report dir to .twd/run', () => { - expect(DEFAULT_REPORT_DIR).toBe('./.twd/run'); - }); -}); -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `npx vitest --run tests/reportFiles.test.js` -Expected: FAIL — `Failed to load ../src/reportFiles.js`. - -- [ ] **Step 3: Write the implementation** - -Create `src/reportFiles.js`: - -```js -import fs from 'fs'; -import path from 'path'; - -export const DEFAULT_REPORT_DIR = './.twd/run'; -export const DEFAULT_MERGED_OUT = './.twd/merged-run.json'; -export const RUN_REPORT_FILE = 'run.json'; -export const COVERAGE_FILE = 'coverage.json'; - -/** - * Writes one shard's report, and its coverage when it collected any. - * - * The report is pretty-printed because a human reads it when a merge complains. - * Coverage is not: it is machine input for nyc, routinely several megabytes, and - * indenting it would roughly double the artifact size for no benefit. - */ -export function writeRunReport(dir, report, coverage = null) { - fs.mkdirSync(dir, { recursive: true }); - - const reportPath = path.join(dir, RUN_REPORT_FILE); - fs.writeFileSync(reportPath, `${JSON.stringify(report, null, 2)}\n`); - - let coveragePath = null; - if (coverage) { - coveragePath = path.join(dir, COVERAGE_FILE); - fs.writeFileSync(coveragePath, JSON.stringify(coverage)); - } - - return { reportPath, coveragePath }; -} - -function readJson(file, label) { - const raw = fs.readFileSync(file, 'utf-8'); - try { - return JSON.parse(raw); - } catch (err) { - throw new Error(`Could not parse ${label} at ${file}: ${err.message}`); - } -} - -/** - * Finds every shard report under `dir`. - * - * actions/download-artifact lays each artifact out as its own directory, so the - * normal shape is `//run.json`. A bare `/run.json` is - * also accepted, which is what a local single-shard run produces. - */ -export function readShardReports(dir) { - const found = []; - - const direct = path.join(dir, RUN_REPORT_FILE); - if (fs.existsSync(direct)) { - found.push({ dir, report: readJson(direct, RUN_REPORT_FILE) }); - } - - let entries; - try { - entries = fs.readdirSync(dir, { withFileTypes: true }); - } catch { - // Missing or unreadable directory: the caller reports "no reports found", - // which is a better message than an ENOENT stack. - return found; - } - - for (const entry of entries) { - if (!entry.isDirectory()) continue; - const shardDir = path.join(dir, entry.name); - const file = path.join(shardDir, RUN_REPORT_FILE); - if (fs.existsSync(file)) { - found.push({ dir: shardDir, report: readJson(file, RUN_REPORT_FILE) }); - } - } - - return found; -} - -/** - * Reads a shard's coverage, or null when it collected none. - * - * Absence is a normal outcome, not an error: a filtered run skips coverage - * entirely. Merge reports how many shards contributed. - */ -export function readShardCoverage(shardDir, coverageFile) { - if (!coverageFile) return null; - const file = path.join(shardDir, coverageFile); - if (!fs.existsSync(file)) return null; - return readJson(file, COVERAGE_FILE); -} -``` - -- [ ] **Step 4: Run the test to verify it passes** - -Run: `npx vitest --run tests/reportFiles.test.js` -Expected: PASS, 11 tests. - -- [ ] **Step 5: Commit** - -```bash -git add src/reportFiles.js tests/reportFiles.test.js -git commit -m "feat(report): read and write shard report artifacts" -``` - ---- - -### Task 5: Merge Istanbul coverage objects - -**Files:** -- Create: `src/mergeCoverage.js` -- Modify: `package.json` (promote `istanbul-lib-coverage` to a runtime dependency) -- Modify: `package-lock.json` (regenerated) -- Test: `tests/mergeCoverage.test.js` - -**Interfaces:** -- Consumes: nothing. -- Produces: `mergeCoverage(coverageObjects: Array) -> object` — an Istanbul coverage map JSON with counts summed across inputs. Nulls are skipped. - -`istanbul-lib-coverage@3.2.2` is currently present only transitively via the `@vitest/coverage-v8` devDependency. It must become a real dependency, because `merge` runs in a consumer's project where devDependencies are not installed. - -**This test file must not mock `fs`** — it exercises a real library against in-memory objects. - -- [ ] **Step 1: Add the dependency** - -```bash -npm install istanbul-lib-coverage@^3.2.2 --save -``` - -Verify it landed under `dependencies` (not `devDependencies`): - -```bash -node -p "require('./package.json').dependencies['istanbul-lib-coverage']" -``` - -Expected: `^3.2.2` - -- [ ] **Step 2: Regenerate the lockfile for Linux** - -Docker must be running. - -```bash -npm run lock:linux -``` - -This is mandatory after any dependency change: npm on macOS never installs the wasm32-wasi optional packages, so it leaves their `@emnapi/*` transitive deps stale in the lock and `npm ci` breaks on Linux CI. - -- [ ] **Step 3: Write the failing test** - -Create `tests/mergeCoverage.test.js`: - -```js -import { describe, it, expect } from 'vitest'; -import { mergeCoverage } from '../src/mergeCoverage.js'; - -// Minimal but structurally valid Istanbul file coverage. istanbul-lib-coverage -// validates the shape, so the maps cannot be omitted. -function fileCoverage(path, statementHits, fnHits = 0) { - return { - path, - statementMap: { 0: { start: { line: 1, column: 0 }, end: { line: 1, column: 10 } } }, - fnMap: { - 0: { - name: 'f', - decl: { start: { line: 1, column: 0 }, end: { line: 1, column: 1 } }, - loc: { start: { line: 1, column: 0 }, end: { line: 3, column: 1 } }, - }, - }, - branchMap: {}, - s: { 0: statementHits }, - f: { 0: fnHits }, - b: {}, - }; -} - -describe('mergeCoverage', () => { - it('sums statement hits for the same file across shards', () => { - const merged = mergeCoverage([ - { '/app/src/a.js': fileCoverage('/app/src/a.js', 1) }, - { '/app/src/a.js': fileCoverage('/app/src/a.js', 2) }, - ]); - expect(merged['/app/src/a.js'].s[0]).toBe(3); - }); - - it('sums function hits for the same file across shards', () => { - const merged = mergeCoverage([ - { '/app/src/a.js': fileCoverage('/app/src/a.js', 1, 4) }, - { '/app/src/a.js': fileCoverage('/app/src/a.js', 1, 5) }, - ]); - expect(merged['/app/src/a.js'].f[0]).toBe(9); - }); - - it('unions files that only one shard touched', () => { - const merged = mergeCoverage([ - { '/app/src/a.js': fileCoverage('/app/src/a.js', 1) }, - { '/app/src/b.js': fileCoverage('/app/src/b.js', 7) }, - ]); - expect(Object.keys(merged).sort()).toEqual(['/app/src/a.js', '/app/src/b.js']); - expect(merged['/app/src/b.js'].s[0]).toBe(7); - }); - - // A shard with no coverage file reads back as null and must not break the merge. - it('skips null and undefined entries', () => { - const merged = mergeCoverage([ - null, - { '/app/src/a.js': fileCoverage('/app/src/a.js', 2) }, - undefined, - ]); - expect(merged['/app/src/a.js'].s[0]).toBe(2); - }); - - it('returns an empty map for no input', () => { - expect(mergeCoverage([])).toEqual({}); - expect(mergeCoverage([null])).toEqual({}); - }); - - it('does not mutate its inputs', () => { - const first = { '/app/src/a.js': fileCoverage('/app/src/a.js', 1) }; - mergeCoverage([first, { '/app/src/a.js': fileCoverage('/app/src/a.js', 5) }]); - expect(first['/app/src/a.js'].s[0]).toBe(1); - }); -}); -``` - -- [ ] **Step 4: Run the test to verify it fails** - -Run: `npx vitest --run tests/mergeCoverage.test.js` -Expected: FAIL — `Failed to load ../src/mergeCoverage.js`. - -- [ ] **Step 5: Write the implementation** - -Create `src/mergeCoverage.js`: - -```js -import libCoverage from 'istanbul-lib-coverage'; - -/** - * Combines per-shard Istanbul coverage into one map. - * - * This is the one artifact that needed no bespoke merge logic: summing hit - * counts across runs of the same code is exactly what CoverageMap.merge does, - * and it is the same operation `nyc merge` performs. - * - * Nulls are skipped rather than rejected — a shard that collected no coverage - * (a filtered run, or one that never loaded instrumented code) reads back as - * null and simply does not contribute. - */ -export function mergeCoverage(coverageObjects) { - const map = libCoverage.createCoverageMap({}); - for (const coverage of coverageObjects) { - if (coverage) map.merge(coverage); - } - return map.toJSON(); -} -``` - -- [ ] **Step 6: Run the test to verify it passes** - -Run: `npx vitest --run tests/mergeCoverage.test.js` -Expected: PASS, 6 tests. - -- [ ] **Step 7: Commit** - -```bash -git add package.json package-lock.json src/mergeCoverage.js tests/mergeCoverage.test.js -git commit -m "feat(coverage): merge per-shard Istanbul coverage maps" -``` - ---- - -### Task 6: Associative report merge with consistency validation - -**Files:** -- Create: `src/mergeReports.js` -- Test: `tests/mergeReports.test.js` - -**Interfaces:** -- Consumes: report objects produced by `buildRunReport` (Task 3). -- Produces: `mergeRunReports(reports: object[]) -> object` (same shape as one report; throws on inconsistency); `findMissingShards(report) -> number[]`; `reportTimings(report) -> { wallMs: number, computeMs: number }`; `reportTotals(report) -> { executed: number, notRun: number, consistent: boolean }`. - -**Critical design point:** completeness (`1..total` all present) is deliberately **not** checked here. A 2-of-3 merge is a legal intermediate value, and rejecting it would make `mergeRunReports([mergeRunReports([a, b]), c])` throw — destroying associativity, which is the property that proves no test is lost or doubled. `findMissingShards` is exported separately and called by the merge *command* (Task 9). - -- [ ] **Step 1: Write the failing test** - -Create `tests/mergeReports.test.js`: - -```js -import { describe, it, expect } from 'vitest'; -import { - mergeRunReports, - findMissingShards, - reportTimings, - reportTotals, -} from '../src/mergeReports.js'; - -const HANDLERS = [ - { id: 's1', name: 'Login', parent: null, type: 'suite' }, - { id: 't1', name: 'a', parent: 's1', type: 'test' }, - { id: 't2', name: 'b', parent: 's1', type: 'test' }, - { id: 't3', name: 'c', parent: 's1', type: 'test' }, -]; - -const FINGERPRINT = 'sha256:deadbeef'; - -function makeReport(index, overrides = {}) { - const { - total = 3, - tests = [{ id: `t${index}`, status: 'pass' }], - startedAt = `2026-08-19T10:00:0${index}.000Z`, - endedAt = `2026-08-19T10:00:1${index}.000Z`, - durationMs = 10_000, - executed = 1, - notRun = 0, - failed = 0, - stoppedEarly = false, - coverageFile = 'coverage.json', - contracts = { configured: true, partial: false, results: [], skipped: [] }, - fingerprint = FINGERPRINT, - schemaVersion = 1, - totalTests = 3, - } = overrides; - - return { - schemaVersion, - shards: [{ - index, total, startedAt, endedAt, durationMs, - executed, notRun, failed, stoppedEarly, coverageFile, recording: null, - }], - discovery: { totalTests, fingerprint }, - selection: { filters: [] }, - handlers: HANDLERS, - tests, - contracts, - }; -} - -describe('mergeRunReports', () => { - it('concatenates tests across shards', () => { - const merged = mergeRunReports([makeReport(1), makeReport(2), makeReport(3)]); - expect(merged.tests.map((t) => t.id)).toEqual(['t1', 't2', 't3']); - }); - - it('sorts shard descriptors by index regardless of input order', () => { - const merged = mergeRunReports([makeReport(3), makeReport(1), makeReport(2)]); - expect(merged.shards.map((s) => s.index)).toEqual([1, 2, 3]); - }); - - it('keeps the single-report shape', () => { - const merged = mergeRunReports([makeReport(1), makeReport(2), makeReport(3)]); - expect(Object.keys(merged).sort()).toEqual( - ['contracts', 'discovery', 'handlers', 'schemaVersion', 'selection', 'shards', 'tests'], - ); - expect(merged.handlers).toEqual(HANDLERS); - expect(merged.discovery).toEqual({ totalTests: 3, fingerprint: FINGERPRINT }); - }); - - // The property that proves nothing is lost or doubled. It only holds because - // completeness is checked outside this function. - it('is associative', () => { - const a = makeReport(1); - const b = makeReport(2); - const c = makeReport(3); - expect(mergeRunReports([mergeRunReports([a, b]), c])) - .toEqual(mergeRunReports([a, b, c])); - expect(mergeRunReports([a, mergeRunReports([b, c])])) - .toEqual(mergeRunReports([a, b, c])); - }); - - it('accepts a partial merge without complaining about gaps', () => { - const merged = mergeRunReports([makeReport(1), makeReport(2)]); - expect(merged.shards.map((s) => s.index)).toEqual([1, 2]); - }); - - it('concatenates contract results and skipped entries', () => { - const merged = mergeRunReports([ - makeReport(1, { contracts: { configured: true, partial: false, results: [{ alias: 'a' }], skipped: [{ alias: 'x' }] } }), - makeReport(2, { contracts: { configured: true, partial: false, results: [{ alias: 'b' }], skipped: [] } }), - ]); - expect(merged.contracts.results).toEqual([{ alias: 'a' }, { alias: 'b' }]); - expect(merged.contracts.skipped).toEqual([{ alias: 'x' }]); - }); - - it('ORs the contracts partial flag', () => { - const partial = makeReport(2, { contracts: { configured: true, partial: true, results: [], skipped: [] } }); - expect(mergeRunReports([makeReport(1), partial]).contracts.partial).toBe(true); - expect(mergeRunReports([makeReport(1), makeReport(2)]).contracts.partial).toBe(false); - }); - - it('throws on an empty input', () => { - expect(() => mergeRunReports([])).toThrow(/No shard reports/); - }); - - it('throws when schema versions disagree', () => { - expect(() => mergeRunReports([makeReport(1), makeReport(2, { schemaVersion: 2 })])) - .toThrow(/schemaVersion/); - }); - - // The safety net: shards that saw different test sets must never be combined. - it('throws when fingerprints disagree', () => { - expect(() => mergeRunReports([makeReport(1), makeReport(2, { fingerprint: 'sha256:other' })])) - .toThrow(/different test sets/); - }); - - it('throws when shard totals disagree', () => { - expect(() => mergeRunReports([makeReport(1), makeReport(2, { total: 4 })])) - .toThrow(/shard total/); - }); - - it('throws when the same shard index appears twice', () => { - expect(() => mergeRunReports([makeReport(1), makeReport(1)])) - .toThrow(/more than once/); - }); - - it('throws when a test id appears in two shards', () => { - expect(() => mergeRunReports([ - makeReport(1, { tests: [{ id: 'dup', status: 'pass' }] }), - makeReport(2, { tests: [{ id: 'dup', status: 'pass' }] }), - ])).toThrow(/"dup" appears in more than one shard/); - }); - - it('does not mutate the input reports', () => { - const a = makeReport(1); - const b = makeReport(2); - mergeRunReports([b, a]); - expect(a.shards).toHaveLength(1); - expect(b.shards[0].index).toBe(2); - }); -}); - -describe('findMissingShards', () => { - it('returns an empty array when every shard is present', () => { - expect(findMissingShards(mergeRunReports([makeReport(1), makeReport(2), makeReport(3)]))) - .toEqual([]); - }); - - it('names the gaps', () => { - expect(findMissingShards(mergeRunReports([makeReport(1), makeReport(3)]))).toEqual([2]); - expect(findMissingShards(mergeRunReports([makeReport(2)]))).toEqual([1, 3]); - }); -}); - -describe('reportTimings', () => { - // Wall clock is what the developer waited; compute is what was paid for. - it('reports wall clock as the span and compute as the sum', () => { - const merged = mergeRunReports([makeReport(1), makeReport(2), makeReport(3)]); - // starts 10:00:01..03, ends 10:00:11..13 -> span 12s; 3 x 10s compute - expect(reportTimings(merged)).toEqual({ wallMs: 12_000, computeMs: 30_000 }); - }); - - it('makes wall and compute equal for a single shard', () => { - const single = mergeRunReports([makeReport(1, { total: 1 })]); - const { wallMs, computeMs } = reportTimings(single); - expect(wallMs).toBe(computeMs); - }); -}); - -describe('reportTotals', () => { - it('sums executed and notRun and confirms they account for discovery', () => { - const merged = mergeRunReports([makeReport(1), makeReport(2), makeReport(3)]); - expect(reportTotals(merged)).toEqual({ executed: 3, notRun: 0, consistent: true }); - }); - - it('flags totals that do not add up to the discovered count', () => { - const merged = mergeRunReports([makeReport(1), makeReport(2)]); - expect(reportTotals(merged).consistent).toBe(false); - }); - - it('counts a bailed shard\'s notRun', () => { - const merged = mergeRunReports([ - makeReport(1), - makeReport(2, { executed: 1, notRun: 1, stoppedEarly: true, failed: 1 }), - makeReport(3), - ]); - expect(reportTotals(merged)).toEqual({ executed: 3, notRun: 1, consistent: false }); - }); -}); -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `npx vitest --run tests/mergeReports.test.js` -Expected: FAIL — `Failed to load ../src/mergeReports.js`. - -- [ ] **Step 3: Write the implementation** - -Create `src/mergeReports.js`: - -```js -/** - * Combines shard reports into one report of the same shape. - * - * Only *consistency* is validated here — the things that stay true under a - * partial merge. Completeness (is 1..total all present?) is checked by - * findMissingShards, called from the merge command, because a 2-of-3 merge is a - * legal intermediate value: rejecting it here would make - * mergeRunReports([mergeRunReports([a, b]), c]) throw and destroy - * associativity, which is the property that proves no test is lost or doubled. - */ -export function mergeRunReports(reports) { - if (!reports.length) { - throw new Error( - 'No shard reports to merge. Expected /*/run.json or /run.json.' - ); - } - - const versions = [...new Set(reports.map((r) => r.schemaVersion))]; - if (versions.length > 1) { - throw new Error( - `Shard reports disagree on schemaVersion (${versions.sort().join(', ')}). ` + - 'Every shard job must run the same twd-cli version.' - ); - } - - const fingerprints = new Set(reports.map((r) => r.discovery.fingerprint)); - if (fingerprints.size > 1) { - throw new Error( - 'Shard reports discovered different test sets, so they cannot be merged. ' + - 'Either tests are registered conditionally (a feature flag, a date, ' + - 'Math.random), or the shard jobs did not build the same code.' - ); - } - - const shards = reports.flatMap((r) => r.shards); - - const totals = [...new Set(shards.map((s) => s.total))]; - if (totals.length > 1) { - throw new Error( - `Shard reports disagree on shard total (${totals.sort((a, b) => a - b).join(', ')}). ` + - 'Every shard job must pass the same --shard total.' - ); - } - - const byIndex = new Set(); - for (const shard of shards) { - if (byIndex.has(shard.index)) { - throw new Error(`Shard ${shard.index}/${shard.total} appears more than once.`); - } - byIndex.add(shard.index); - } - - const tests = []; - const testIds = new Set(); - for (const report of reports) { - for (const test of report.tests) { - if (testIds.has(test.id)) { - throw new Error( - `Test id "${test.id}" appears in more than one shard — the shard slices overlap.` - ); - } - testIds.add(test.id); - tests.push(test); - } - } - - const first = reports[0]; - - return { - schemaVersion: first.schemaVersion, - // Copy before sorting: sort mutates, and the input reports are the caller's. - shards: [...shards].sort((a, b) => a.index - b.index), - discovery: first.discovery, - selection: first.selection, - // Identical across shards, guaranteed by the fingerprint check above. - handlers: first.handlers, - tests, - contracts: { - configured: first.contracts.configured, - partial: reports.some((r) => r.contracts.partial), - results: reports.flatMap((r) => r.contracts.results), - skipped: reports.flatMap((r) => r.contracts.skipped), - }, - }; -} - -/** - * Shard indices in 1..total that no report accounted for. - * - * A gap almost always means a shard job died before uploading its artifact. It - * must be loud: a silent 3-of-4 merge reads as a complete green run. - */ -export function findMissingShards(report) { - const total = report.shards[0]?.total ?? 0; - const present = new Set(report.shards.map((s) => s.index)); - const missing = []; - for (let i = 1; i <= total; i++) { - if (!present.has(i)) missing.push(i); - } - return missing; -} - -/** - * Wall clock (what the developer waited) and compute (what was paid for). - * - * Derived rather than stored, so the two can never drift out of agreement with - * the per-shard timestamps they come from. - */ -export function reportTimings(report) { - const starts = report.shards.map((s) => Date.parse(s.startedAt)); - const ends = report.shards.map((s) => Date.parse(s.endedAt)); - return { - wallMs: Math.max(...ends) - Math.min(...starts), - computeMs: report.shards.reduce((sum, s) => sum + s.durationMs, 0), - }; -} - -/** - * Executed and not-run totals, plus whether they account for every discovered - * test. An inconsistent result points at a shard-math bug, not user error. - */ -export function reportTotals(report) { - const executed = report.shards.reduce((sum, s) => sum + s.executed, 0); - const notRun = report.shards.reduce((sum, s) => sum + s.notRun, 0); - return { - executed, - notRun, - consistent: executed + notRun === report.discovery.totalTests, - }; -} -``` - -- [ ] **Step 4: Run the test to verify it passes** - -Run: `npx vitest --run tests/mergeReports.test.js` -Expected: PASS, 22 tests. - -- [ ] **Step 5: Commit** - -```bash -git add src/mergeReports.js tests/mergeReports.test.js -git commit -m "feat(merge): associative report merge with consistency validation" -``` - ---- - -### Task 7: Per-shard breakdown in the run summary - -**Files:** -- Modify: `src/testSummary.js` -- Test: `tests/testSummary.test.js` - -**Interfaces:** -- Consumes: `shards` array from a merged report (Task 6), `computeMs` from `reportTimings`. -- Produces: `formatRunComplete({ testStatus, handlers, durationMs, notRun, stoppedEarly, maxFailures, shards, computeMs })` — two new optional params, both defaulting to `null`. - -**Constraint:** with `shards` absent or of length 1, output must be byte-identical to today's. - -- [ ] **Step 1: Write the failing test** - -Append to `tests/testSummary.test.js`: - -```js -describe('formatRunComplete with shards', () => { - const handlers = [ - { id: 's1', name: 'Login', parent: null, type: 'suite' }, - { id: 't1', name: 'works', parent: 's1', type: 'test' }, - ]; - const testStatus = [{ id: 't1', status: 'pass' }]; - - function shard(index, overrides = {}) { - return { index, total: 4, executed: 30, failed: 0, notRun: 0, stoppedEarly: false, ...overrides }; - } - - it('adds a shard breakdown line when more than one shard merged', () => { - const output = formatRunComplete({ - testStatus, handlers, durationMs: 38_200, computeMs: 134_200, - shards: [shard(1), shard(2, { failed: 3 }), shard(3), shard(4)], - }); - expect(output).toContain('Shards: 1 ✓30 | 2 ✗30 | 3 ✓30 | 4 ✓30'); - }); - - it('reports wall clock and compute separately for a merged run', () => { - const output = formatRunComplete({ - testStatus, handlers, durationMs: 38_200, computeMs: 134_200, - shards: [shard(1), shard(2)], - }); - expect(output).toContain('Duration: 38.2s wall | 134.2s across 2 shards'); - }); - - // The existing single-run format must not shift. - it('keeps the plain duration line when there are no shards', () => { - const output = formatRunComplete({ testStatus, handlers, durationMs: 4_200 }); - expect(output).toContain('Duration: 4.2s'); - expect(output).not.toContain('wall'); - expect(output).not.toContain('Shards:'); - }); - - it('keeps the plain duration line for a single shard', () => { - const output = formatRunComplete({ - testStatus, handlers, durationMs: 4_200, computeMs: 4_200, shards: [shard(1, { total: 1 })], - }); - expect(output).toContain('Duration: 4.2s'); - expect(output).not.toContain('Shards:'); - }); -}); -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `npx vitest --run tests/testSummary.test.js` -Expected: FAIL — the breakdown and wall/compute assertions find no such line. - -- [ ] **Step 3: Implement the breakdown** - -In `src/testSummary.js`, extend the destructured parameters: - -```js -export function formatRunComplete({ - testStatus, - handlers, - durationMs, - notRun = 0, - stoppedEarly = false, - maxFailures, - shards = null, - computeMs = null, -}) { -``` - -Replace the existing duration block: - -```js - if (notRun > 0) lines.push(` Not run: ${notRun}`); - lines.push(` Duration: ${duration}s`); -``` - -with: - -```js - if (notRun > 0) lines.push(` Not run: ${notRun}`); - - // A merged run has two meaningful durations: the span the developer waited, - // and the compute it consumed. A single run has only one, and its line must - // stay byte-identical to what it has always printed. - const merged = Array.isArray(shards) && shards.length > 1; - if (merged) { - const compute = (computeMs / 1000).toFixed(1); - lines.push(` Duration: ${duration}s wall | ${compute}s across ${shards.length} shards`); - const cells = shards.map((s) => `${s.index} ${s.failed > 0 ? '✗' : '✓'}${s.executed}`); - lines.push(` Shards: ${cells.join(' | ')}`); - } else { - lines.push(` Duration: ${duration}s`); - } -``` - -- [ ] **Step 4: Run the full suite to verify nothing shifted** - -Run: `npx vitest --run tests/testSummary.test.js` -Expected: PASS — the new tests plus every existing one, unchanged. - -- [ ] **Step 5: Commit** - -```bash -git add src/testSummary.js tests/testSummary.test.js -git commit -m "feat(summary): add per-shard breakdown and wall vs compute duration" -``` - ---- - -### Task 8: Wire sharding and report writing into the run - -**Files:** -- Modify: `src/index.js` -- Test: `tests/runTests.test.js` - -**Interfaces:** -- Consumes: `selectShardIds` (Task 1), `buildRunReport` (Task 3), `writeRunReport` / `DEFAULT_REPORT_DIR` / `COVERAGE_FILE` (Task 4). -- Produces: `runTests({ testFilters, recordOverrides, shard, reportDir }) -> Promise`. Two new options; the return value is unchanged. - -This is the task where the Global Constraint bites hardest. With `shard` absent, every conditional below must reduce to the expression it replaced. - -- [ ] **Step 1: Write the failing tests** - -Append to `tests/runTests.test.js`: - -```js -describe('runTests sharding', () => { - function fourTests() { - return { - handlers: [ - { id: '1', name: 'a', type: 'test' }, - { id: '2', name: 'b', type: 'test' }, - { id: '3', name: 'c', type: 'test' }, - { id: '4', name: 'd', type: 'test' }, - ], - testStatus: [{ id: '2', status: 'pass' }, { id: '4', status: 'pass' }], - }; - } - - function runJson() { - const call = vi.mocked(fs.writeFileSync).mock.calls - .find(([file]) => String(file).endsWith('run.json')); - return call ? JSON.parse(call[1]) : null; - } - - // Round-robin: shard 2 of 2 takes the odd indices, i.e. the 2nd and 4th ids. - it('runs only its round-robin slice', async () => { - const { handlers, testStatus } = fourTests(); - const page = createMockPage({ handlers, testStatus }); - vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); - - await runTests({ shard: { index: 2, total: 2 } }); - - expect(page.evaluate).toHaveBeenCalledWith(expect.any(Function), 2, ['2', '4']); - }); - - it('writes a run report to the default report dir', async () => { - const { handlers, testStatus } = fourTests(); - const page = createMockPage({ handlers, testStatus }); - vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); - - await runTests({ shard: { index: 2, total: 2 } }); - - expect(fs.mkdirSync).toHaveBeenCalledWith('./.twd/run', { recursive: true }); - const report = runJson(); - expect(report.schemaVersion).toBe(1); - expect(report.shards[0]).toMatchObject({ index: 2, total: 2, executed: 2 }); - expect(report.discovery.totalTests).toBe(4); - expect(report.tests.map((t) => t.id)).toEqual(['2', '4']); - }); - - it('honors --report-dir', async () => { - const { handlers, testStatus } = fourTests(); - const page = createMockPage({ handlers, testStatus }); - vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); - - await runTests({ shard: { index: 1, total: 1 }, reportDir: './out' }); - - expect(fs.mkdirSync).toHaveBeenCalledWith('./out', { recursive: true }); - }); - - it('writes no report when not sharded', async () => { - const { handlers, testStatus } = fourTests(); - const page = createMockPage({ handlers, testStatus }); - vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); - - await runTests(); - - expect(runJson()).toBeNull(); - }); - - // 3 tests across 4 shards leaves the fourth with nothing to run. It must - // still write a valid report, or merge sees a gap it cannot explain. - it('writes a valid empty report when its slice is empty', async () => { - const handlers = [ - { id: '1', name: 'a', type: 'test' }, - { id: '2', name: 'b', type: 'test' }, - { id: '3', name: 'c', type: 'test' }, - ]; - const page = createMockPage({ handlers, testStatus: [] }); - vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); - - const hasFailures = await runTests({ shard: { index: 4, total: 4 } }); - - expect(hasFailures).toBe(false); - const report = runJson(); - expect(report.tests).toEqual([]); - expect(report.shards[0]).toMatchObject({ index: 4, total: 4, executed: 0, failed: 0 }); - // The fingerprint still covers the whole suite, so an empty shard merges - // cleanly with the three that ran something. - expect(report.discovery.totalTests).toBe(3); - }); - - // Filters resolve first, then the filtered list is sharded. A filtered run's - // coverage is a misleading project-wide number, sharded or not. - it('skips coverage when a filter is combined with a shard', async () => { - const { handlers, testStatus } = fourTests(); - const page = createMockPage({ handlers, testStatus }); - vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); - vi.mocked(loadConfig).mockReturnValue({ ...defaultMockConfig, coverage: true }); - - await runTests({ shard: { index: 1, total: 2 }, testFilters: ['a'] }); - - const files = vi.mocked(fs.writeFileSync).mock.calls.map(([f]) => String(f)); - expect(files.some((f) => f.endsWith('coverage.json'))).toBe(false); - expect(runJson().shards[0].coverageFile).toBeNull(); - expect(runJson().selection.filters).toEqual(['a']); - }); -}); - -describe('runTests non-regression: non-sharded behavior is unchanged', () => { - // The !hasFailures coverage gate is relaxed only for sharded runs. A plain - // failing run must still write nothing, exactly as in 1.4.0. - it('writes no coverage when a non-sharded run fails', async () => { - const handlers = [{ id: '1', name: 'a', type: 'test' }]; - const page = createMockPage({ handlers, testStatus: [{ id: '1', status: 'fail', error: 'boom' }] }); - vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); - vi.mocked(loadConfig).mockReturnValue({ ...defaultMockConfig, coverage: true }); - - await runTests(); - - expect(fs.writeFileSync).not.toHaveBeenCalled(); - }); - - // Likewise the early-stop contract skip. - it('still skips contract validation when a non-sharded run stops early', async () => { - const handlers = [{ id: '1', name: 'a', type: 'test' }]; - const page = createMockPage({ handlers, testStatus: [{ id: '1', status: 'fail', error: 'boom' }] }); - vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); - vi.mocked(loadConfig).mockReturnValue({ - ...defaultMockConfig, maxFailures: 1, contracts: [{ source: 'api.json' }], - }); - vi.mocked(loadContracts).mockResolvedValue([]); - - await runTests(); - - expect(validateMocks).not.toHaveBeenCalled(); - }); -}); - -describe('runTests sharded behavior changes', () => { - it('writes coverage for a sharded run that fails', async () => { - const handlers = [{ id: '1', name: 'a', type: 'test' }]; - const page = createMockPage({ handlers, testStatus: [{ id: '1', status: 'fail', error: 'boom' }] }); - vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); - vi.mocked(loadConfig).mockReturnValue({ ...defaultMockConfig, coverage: true }); - - await runTests({ shard: { index: 1, total: 1 } }); - - const files = vi.mocked(fs.writeFileSync).mock.calls.map(([f]) => String(f)); - expect(files.some((f) => f.endsWith('coverage.json'))).toBe(true); - // Never the path nyc reads by default: a shard's partial coverage there - // would masquerade as the whole run's. - expect(files.some((f) => f.includes('.nyc_output'))).toBe(false); - }); - - it('validates contracts on a sharded early stop and marks them partial', async () => { - const handlers = [{ id: '1', name: 'a', type: 'test' }]; - const page = createMockPage({ handlers, testStatus: [{ id: '1', status: 'fail', error: 'boom' }] }); - vi.mocked(puppeteer.launch).mockResolvedValue(createMockBrowser(page)); - vi.mocked(loadConfig).mockReturnValue({ - ...defaultMockConfig, maxFailures: 1, contracts: [{ source: 'api.json' }], - }); - vi.mocked(loadContracts).mockResolvedValue([]); - vi.mocked(validateMocks).mockReturnValue({ results: [{ alias: 'a' }], skipped: [] }); - vi.mocked(printContractReport).mockReturnValue(false); - - await runTests({ shard: { index: 1, total: 1 } }); - - expect(validateMocks).toHaveBeenCalled(); - const call = vi.mocked(fs.writeFileSync).mock.calls - .find(([file]) => String(file).endsWith('run.json')); - const report = JSON.parse(call[1]); - expect(report.contracts).toMatchObject({ configured: true, partial: true }); - expect(report.contracts.results).toEqual([{ alias: 'a' }]); - }); -}); -``` - -- [ ] **Step 2: Run the tests to verify they fail** - -Run: `npx vitest --run tests/runTests.test.js` -Expected: FAIL — the shard slice assertion sees all four ids, and no `run.json` is written. - -- [ ] **Step 3: Add imports and options** - -In `src/index.js`, add to the import block: - -```js -import { selectShardIds } from './shard.js'; -import { buildRunReport } from './runReport.js'; -import { writeRunReport, DEFAULT_REPORT_DIR, COVERAGE_FILE } from './reportFiles.js'; -``` - -Change the destructure at line 46: - -```js - const { testFilters = [], recordOverrides = {}, shard = null, reportDir = null } = options; - const sharded = Boolean(shard); -``` - -Add one declaration alongside the other `let`s near the top of the function, so the recording details can reach the report: - -```js - let recordingInfo = null; -``` - -- [ ] **Step 4: Slice the ids** - -Replace line 164: - -```js - const baseIds = selectedIds ?? orderedTestIds(registeredHandlers); -``` - -with: - -```js - // The full ordered list, before filtering or slicing. This is what the - // fingerprint hashes and what discovery.totalTests reports, so every shard - // agrees on it regardless of which slice it took. - const allTestIds = orderedTestIds(registeredHandlers); - const filteredIds = selectedIds ?? allTestIds; - const baseIds = sharded - ? selectShardIds(filteredIds, shard.index, shard.total) - : filteredIds; - - if (sharded) { - console.log( - `Shard ${shard.index}/${shard.total}: running ${baseIds.length} of ${filteredIds.length} test(s).` - ); - } -``` - -- [ ] **Step 5: Capture recording details for the report** - -In the recording success branch (around line 277), replace: - -```js - } else { - console.log(`Recorded ${executed} test(s) to ${recordOutput}`); - } -``` - -with: - -```js - } else { - recordingInfo = { file: recordOutput, bytes: recordedFileSize(recordOutputPath) }; - console.log(`Recorded ${executed} test(s) to ${recordOutput}`); - } -``` - -- [ ] **Step 6: Capture the end timestamp** - -Replace line 282: - -```js - const durationMs = Date.now() - startedAt; -``` - -with: - -```js - const endedAt = Date.now(); - const durationMs = endedAt - startedAt; -``` - -- [ ] **Step 7: Gate the contract change on sharding** - -Replace the whole contract block (lines 296-319) with: - -```js - // Contract validation. A sharded run validates even after an early stop and - // flags the result partial, so merge can say exactly what is missing rather - // than silently dropping a shard's worth of mocks. A non-sharded run keeps - // skipping, exactly as before. - const contractsConfigured = Boolean(config.contracts && config.contracts.length > 0); - let contractsBlock = { - configured: contractsConfigured, - partial: false, - results: [], - skipped: [], - }; - - if (contractsConfigured && (sharded || !stoppedEarly)) { - if (collectedMocks.size === 0) { - console.log('\nNo mocks collected — ensure twd-js supports contract collection'); - } - const validationOutput = validateMocks(collectedMocks, contractValidators); - const hasContractErrors = printContractReport(validationOutput); - if (hasContractErrors) { - hasFailures = true; - } - - contractsBlock = { - configured: true, - partial: stoppedEarly, - results: validationOutput.results, - skipped: validationOutput.skipped, - }; - - if (stoppedEarly) { - console.log('\n⚠ Contract data is partial — this shard stopped early.'); - } - - // Only a whole run produces a meaningful markdown report. Under sharding - // each shard would overwrite the others with a quarter of the picture, so - // `merge` writes it instead. - if (config.contractReportPath && !sharded) { - const reportPath = path.resolve(workingDir, config.contractReportPath); - const reportDirPath = path.dirname(reportPath); - if (!fs.existsSync(reportDirPath)) { - fs.mkdirSync(reportDirPath, { recursive: true }); - } - const markdown = generateContractMarkdown(validationOutput); - fs.writeFileSync(reportPath, markdown); - console.log(`Contract report written to ${config.contractReportPath}`); - } - } else if (contractsConfigured && stoppedEarly) { - console.log('\nSkipping contract validation — run stopped early (partial data).'); - } -``` - -- [ ] **Step 8: Gate the coverage change on sharding** - -Replace the whole coverage block (lines 321-344) with: - -```js - // Handle code coverage. - // - // The filter gate is unchanged: a --test filter still suppresses coverage, - // because a filtered run's number is a misleading project-wide figure. A - // shard slice is not a filter. - // - // The failure gate is relaxed for sharded runs only. hasFailures is per - // shard, so applying it here would let three green shards write coverage - // while a red fourth writes none — a merged report that looks complete but - // is missing a quarter of the code paths. `merge` applies the gate to the - // true global result instead. - if (selectedIds && config.coverage) { - console.log('Skipping coverage collection (test filter active).'); - } - - let coverageData = null; - if (config.coverage && !selectedIds && (sharded || !hasFailures)) { - coverageData = await page.evaluate(() => window.__coverage__); - if (!coverageData) { - console.log('No code coverage data found.'); - } - } - - // A sharded run's coverage goes to the report dir and nowhere else. Writing - // it to .nyc_output/out.json — the path nyc reads by default — would let one - // shard's partial data masquerade as the whole run's. - if (coverageData && !sharded) { - const coverageDir = path.resolve(workingDir, config.coverageDir); - const nycDir = path.resolve(workingDir, config.nycOutputDir); - - if (!fs.existsSync(nycDir)) { - fs.mkdirSync(nycDir, { recursive: true }); - } - if (!fs.existsSync(coverageDir)) { - fs.mkdirSync(coverageDir, { recursive: true }); - } - - const coveragePath = path.join(nycDir, 'out.json'); - fs.writeFileSync(coveragePath, JSON.stringify(coverageData)); - console.log(`Code coverage data written to ${coveragePath}`); - } -``` - -- [ ] **Step 9: Write the shard report** - -After the `formatRunComplete` block (line 357) and before `return hasFailures;`, insert: - -```js - // Written last, and only for a sharded run. A run that threw never gets - // here on purpose: its artifact stays absent, and `merge` reports the gap as - // "a shard job likely failed before uploading", which is the accurate - // diagnosis. A half-written report would be a worse lie. - if (sharded) { - const dir = reportDir ?? DEFAULT_REPORT_DIR; - const report = buildRunReport({ - shard, - startedAt, - endedAt, - allTestIds, - filters: testFilters, - handlers, - tests: testStatus, - executed, - notRun, - stoppedEarly, - coverageFile: coverageData ? COVERAGE_FILE : null, - recording: recordingInfo, - contracts: contractsBlock, - }); - const { reportPath } = writeRunReport(dir, report, coverageData); - console.log(`Shard report written to ${reportPath}`); - } -``` - -- [ ] **Step 10: Run the full suite** - -Run: `npm run test:ci` -Expected: PASS — every existing test plus the new ones. The two non-regression tests are the ones that matter most here. - -- [ ] **Step 11: Commit** - -```bash -git add src/index.js tests/runTests.test.js -git commit -m "feat(run): slice tests by shard and write a run report artifact" -``` - ---- - -### Task 9: The `merge` command - -**Files:** -- Create: `src/mergeCommand.js` -- Modify: `bin/twd-cli.js` -- Test: `tests/mergeCommand.test.js` - -**Interfaces:** -- Consumes: `readShardReports` / `readShardCoverage` / `DEFAULT_MERGED_OUT` (Task 4), `mergeCoverage` (Task 5), `mergeRunReports` / `findMissingShards` / `reportTimings` / `reportTotals` (Task 6), `formatRunComplete` (Task 7), plus the existing `loadConfig`, `printContractReport`, `generateContractMarkdown`. -- Produces: `runMerge({ dir, out }) -> boolean` (true when the merged run has failures). Throws on unusable input. - -- [ ] **Step 1: Write the failing test** - -Create `tests/mergeCommand.test.js`: - -```js -import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; - -vi.mock('fs'); -vi.mock('../src/config.js', () => ({ loadConfig: vi.fn() })); -vi.mock('../src/contractReport.js', () => ({ printContractReport: vi.fn() })); -vi.mock('../src/reportFiles.js', async (importOriginal) => { - const actual = await importOriginal(); - return { ...actual, readShardReports: vi.fn(), readShardCoverage: vi.fn() }; -}); - -import fs from 'fs'; -import { loadConfig } from '../src/config.js'; -import { printContractReport } from '../src/contractReport.js'; -import { readShardReports, readShardCoverage } from '../src/reportFiles.js'; -import { runMerge } from '../src/mergeCommand.js'; - -const HANDLERS = [ - { id: 's1', name: 'Login', parent: null, type: 'suite' }, - { id: 't1', name: 'a', parent: 's1', type: 'test' }, - { id: 't2', name: 'b', parent: 's1', type: 'test' }, -]; - -function shardReport(index, overrides = {}) { - const { total = 2, tests = [{ id: `t${index}`, status: 'pass' }], failed = 0 } = overrides; - return { - schemaVersion: 1, - shards: [{ - index, total, - startedAt: `2026-08-19T10:00:0${index}.000Z`, - endedAt: `2026-08-19T10:00:1${index}.000Z`, - durationMs: 10_000, - executed: 1, notRun: 0, failed, - stoppedEarly: false, coverageFile: 'coverage.json', recording: null, - }], - discovery: { totalTests: 2, fingerprint: 'sha256:same' }, - selection: { filters: [] }, - handlers: HANDLERS, - tests, - contracts: { configured: false, partial: false, results: [], skipped: [] }, - }; -} - -const baseConfig = { - coverage: true, - nycOutputDir: './.nyc_output', - maxFailures: 10, -}; - -function writtenFiles() { - return vi.mocked(fs.writeFileSync).mock.calls.map(([f]) => String(f)); -} - -describe('runMerge', () => { - beforeEach(() => { - vi.clearAllMocks(); - vi.mocked(loadConfig).mockReturnValue({ ...baseConfig }); - vi.mocked(readShardCoverage).mockReturnValue(null); - vi.spyOn(console, 'log').mockImplementation(() => {}); - vi.spyOn(console, 'warn').mockImplementation(() => {}); - }); - - afterEach(() => { - vi.restoreAllMocks(); - }); - - it('requires a directory', () => { - expect(() => runMerge({})).toThrow(/Usage: twd-cli merge/); - }); - - it('errors when no shard reports were found', () => { - vi.mocked(readShardReports).mockReturnValue([]); - expect(() => runMerge({ dir: '.twd/shards' })).toThrow(/No shard reports found/); - }); - - // The failure that must never be silent: three green shards and one that - // never uploaded would otherwise read as a complete green run. - it('errors and names the gap when a shard is missing', () => { - vi.mocked(readShardReports).mockReturnValue([{ dir: 'a', report: shardReport(1) }]); - expect(() => runMerge({ dir: '.twd/shards' })).toThrow(/Missing shard report\(s\): 2\/2/); - }); - - it('mentions if: always() in the missing-shard message', () => { - vi.mocked(readShardReports).mockReturnValue([{ dir: 'a', report: shardReport(2) }]); - expect(() => runMerge({ dir: '.twd/shards' })).toThrow(/if: always\(\)/); - }); - - it('writes the merged report to the default path', () => { - vi.mocked(readShardReports).mockReturnValue([ - { dir: 'a', report: shardReport(1) }, - { dir: 'b', report: shardReport(2) }, - ]); - - expect(runMerge({ dir: '.twd/shards' })).toBe(false); - - const merged = writtenFiles().find((f) => f.endsWith('merged-run.json')); - expect(merged).toBeDefined(); - }); - - it('honors --out', () => { - vi.mocked(readShardReports).mockReturnValue([ - { dir: 'a', report: shardReport(1) }, - { dir: 'b', report: shardReport(2) }, - ]); - - runMerge({ dir: '.twd/shards', out: 'custom.json' }); - - expect(writtenFiles().some((f) => f.endsWith('custom.json'))).toBe(true); - }); - - it('returns true when any shard had a failing test', () => { - vi.mocked(readShardReports).mockReturnValue([ - { dir: 'a', report: shardReport(1) }, - { dir: 'b', report: shardReport(2, { tests: [{ id: 't2', status: 'fail', error: 'boom' }], failed: 1 }) }, - ]); - - expect(runMerge({ dir: '.twd/shards' })).toBe(true); - }); - - it('merges coverage when the run is green', () => { - vi.mocked(readShardReports).mockReturnValue([ - { dir: 'a', report: shardReport(1) }, - { dir: 'b', report: shardReport(2) }, - ]); - vi.mocked(readShardCoverage).mockReturnValue({}); - - runMerge({ dir: '.twd/shards' }); - - expect(writtenFiles().some((f) => f.includes('.nyc_output'))).toBe(true); - }); - - // The user's rule, applied to the true global result rather than one shard's. - it('skips merged coverage when the run is red', () => { - vi.mocked(readShardReports).mockReturnValue([ - { dir: 'a', report: shardReport(1) }, - { dir: 'b', report: shardReport(2, { tests: [{ id: 't2', status: 'fail', error: 'boom' }], failed: 1 }) }, - ]); - vi.mocked(readShardCoverage).mockReturnValue({}); - - runMerge({ dir: '.twd/shards' }); - - expect(writtenFiles().some((f) => f.includes('.nyc_output'))).toBe(false); - }); - - it('reports how many shards contributed coverage', () => { - const log = vi.spyOn(console, 'log'); - vi.mocked(readShardReports).mockReturnValue([ - { dir: 'a', report: shardReport(1) }, - { dir: 'b', report: shardReport(2) }, - ]); - vi.mocked(readShardCoverage).mockReturnValueOnce({}).mockReturnValueOnce(null); - - runMerge({ dir: '.twd/shards' }); - - expect(log.mock.calls.flat().join('\n')).toMatch(/Coverage merged from 1\/2 shards/); - }); - - it('says so when no shard had coverage', () => { - const log = vi.spyOn(console, 'log'); - vi.mocked(readShardReports).mockReturnValue([ - { dir: 'a', report: shardReport(1) }, - { dir: 'b', report: shardReport(2) }, - ]); - - runMerge({ dir: '.twd/shards' }); - - expect(log.mock.calls.flat().join('\n')).toMatch(/No coverage data found/); - }); - - it('returns true when contracts report an error-mode violation', () => { - const withContracts = (i) => ({ - ...shardReport(i), - contracts: { configured: true, partial: false, results: [{ alias: 'a' }], skipped: [] }, - }); - vi.mocked(readShardReports).mockReturnValue([ - { dir: 'a', report: withContracts(1) }, - { dir: 'b', report: withContracts(2) }, - ]); - vi.mocked(printContractReport).mockReturnValue(true); - - expect(runMerge({ dir: '.twd/shards' })).toBe(true); - }); - - it('warns when contract data is partial', () => { - const warn = vi.spyOn(console, 'warn'); - const partial = (i, isPartial) => ({ - ...shardReport(i), - contracts: { configured: true, partial: isPartial, results: [], skipped: [] }, - }); - vi.mocked(readShardReports).mockReturnValue([ - { dir: 'a', report: partial(1, false) }, - { dir: 'b', report: partial(2, true) }, - ]); - vi.mocked(printContractReport).mockReturnValue(false); - - runMerge({ dir: '.twd/shards' }); - - expect(warn.mock.calls.flat().join('\n')).toMatch(/contract data is partial/i); - }); - - it('prints the merged run-complete block with a shard breakdown', () => { - const log = vi.spyOn(console, 'log'); - vi.mocked(readShardReports).mockReturnValue([ - { dir: 'a', report: shardReport(1) }, - { dir: 'b', report: shardReport(2) }, - ]); - - runMerge({ dir: '.twd/shards' }); - - const output = log.mock.calls.flat().join('\n'); - expect(output).toContain('--- Run complete ---'); - expect(output).toContain('Shards: 1 ✓1 | 2 ✓1'); - }); -}); -``` - -- [ ] **Step 2: Run the test to verify it fails** - -Run: `npx vitest --run tests/mergeCommand.test.js` -Expected: FAIL — `Failed to load ../src/mergeCommand.js`. - -- [ ] **Step 3: Write the implementation** - -Create `src/mergeCommand.js`: - -```js -import fs from 'fs'; -import path from 'path'; -import { loadConfig } from './config.js'; -import { - readShardReports, - readShardCoverage, - DEFAULT_MERGED_OUT, -} from './reportFiles.js'; -import { - mergeRunReports, - findMissingShards, - reportTimings, - reportTotals, -} from './mergeReports.js'; -import { mergeCoverage } from './mergeCoverage.js'; -import { formatRunComplete } from './testSummary.js'; -import { printContractReport } from './contractReport.js'; -import { generateContractMarkdown } from './contractMarkdown.js'; - -/** - * Joins per-shard reports into one and reports on the whole run. - * - * This function owns the run's exit code. Shard jobs each exit 1 on their own - * failures, so the workflow only reaches here with `if: !cancelled()`, and the - * merged verdict is the one that counts. - */ -export function runMerge({ dir, out = null } = {}) { - if (!dir) { - throw new Error('Usage: twd-cli merge [--out ]'); - } - - const config = loadConfig(); - const workingDir = process.cwd(); - - const found = readShardReports(dir); - if (found.length === 0) { - throw new Error( - `No shard reports found in ${dir}. ` + - 'Expected /*/run.json (the actions/download-artifact layout) or /run.json.' - ); - } - - const merged = mergeRunReports(found.map((f) => f.report)); - - // Completeness is enforced here rather than inside mergeRunReports, which must - // stay associative. A gap is never a warning: a silent 3-of-4 merge reads as a - // complete green run. - const missing = findMissingShards(merged); - if (missing.length > 0) { - const total = merged.shards[0].total; - throw new Error( - `Missing shard report(s): ${missing.map((i) => `${i}/${total}`).join(', ')}. ` + - 'A shard job likely failed before uploading its artifact — check that the ' + - 'upload step runs with `if: always()`.' - ); - } - - const outPath = path.resolve(workingDir, out ?? DEFAULT_MERGED_OUT); - fs.mkdirSync(path.dirname(outPath), { recursive: true }); - fs.writeFileSync(outPath, `${JSON.stringify(merged, null, 2)}\n`); - console.log(`Merged report written to ${outPath}`); - - let hasFailures = merged.tests.some((test) => test.status === 'fail'); - - if (merged.contracts.configured) { - const validationOutput = { - results: merged.contracts.results, - skipped: merged.contracts.skipped, - }; - if (printContractReport(validationOutput)) { - hasFailures = true; - } - if (merged.contracts.partial) { - console.warn( - 'Warning: contract data is partial — at least one shard stopped early, so ' + - 'some mocks were never collected.' - ); - } - if (config.contractReportPath) { - const reportPath = path.resolve(workingDir, config.contractReportPath); - fs.mkdirSync(path.dirname(reportPath), { recursive: true }); - fs.writeFileSync(reportPath, generateContractMarkdown(validationOutput)); - console.log(`Contract report written to ${config.contractReportPath}`); - } - } - - // A red run yields no coverage — the same policy a single run has always had, - // but keyed on the whole merged result instead of one shard's. - if (config.coverage) { - const coverages = found.map((f) => - readShardCoverage(f.dir, f.report.shards[0]?.coverageFile ?? null) - ); - const contributors = coverages.filter(Boolean).length; - - if (contributors === 0) { - console.log('No coverage data found in any shard.'); - } else if (hasFailures) { - console.log( - `Skipping merged coverage — the run has failures ` + - `(${contributors}/${found.length} shard(s) had data).` - ); - } else { - const nycDir = path.resolve(workingDir, config.nycOutputDir); - fs.mkdirSync(nycDir, { recursive: true }); - fs.writeFileSync(path.join(nycDir, 'out.json'), JSON.stringify(mergeCoverage(coverages))); - console.log( - `Coverage merged from ${contributors}/${found.length} shards to ` + - `${config.nycOutputDir}/out.json` - ); - } - } - - const totals = reportTotals(merged); - if (!totals.consistent) { - console.warn( - `Warning: shard totals do not add up — ${totals.executed} executed + ` + - `${totals.notRun} not run != ${merged.discovery.totalTests} discovered. ` + - 'This points at a shard-slicing bug, not at your tests.' - ); - } - - const timings = reportTimings(merged); - console.log(''); - console.log(formatRunComplete({ - testStatus: merged.tests, - handlers: merged.handlers, - durationMs: timings.wallMs, - computeMs: timings.computeMs, - notRun: totals.notRun, - shards: merged.shards, - stoppedEarly: merged.shards.some((s) => s.stoppedEarly), - maxFailures: config.maxFailures, - })); - - return hasFailures; -} -``` - -- [ ] **Step 4: Run the test to verify it passes** - -Run: `npx vitest --run tests/mergeCommand.test.js` -Expected: PASS, 15 tests. - -- [ ] **Step 5: Wire the subcommand into the CLI** - -In `bin/twd-cli.js`, extend the imports: - -```js -import { runTests } from '../src/index.js'; -import { parseRunArgs, parseMergeArgs } from '../src/parseArgs.js'; -import { runMerge } from '../src/mergeCommand.js'; -``` - -Change the `run` branch to forward the new options: - -```js - const { testFilters, record, shard, reportDir } = parseRunArgs(process.argv.slice(3)); - const hasFailures = await runTests({ - testFilters, - recordOverrides: record, - shard, - reportDir, - }); -``` - -Add a `merge` branch immediately after the `run` block's closing brace: - -```js -} else if (command === 'merge') { - try { - const { dir, out } = parseMergeArgs(process.argv.slice(3)); - const hasFailures = runMerge({ dir, out }); - process.exit(hasFailures ? 1 : 0); - } catch (error) { - if (!error?.reported) { - console.error(error?.message ?? String(error)); - } - process.exit(1); - } -} else { -``` - -- [ ] **Step 6: Update the help text** - -In the same file's help block, add to `Usage:`: - -``` - npx twd-cli run --shard 2/4 Run only this shard's slice of the suite - and write a report to ./.twd/run - npx twd-cli merge Merge shard reports from into one - report, and exit 1 if the whole run failed -``` - -Add to `Options:`: - -``` - --shard / Run slice i of n. Each shard discovers the whole - suite and takes every nth test, so the test count - never has to be known in advance. Implies a report. - --report-dir Where to write the shard report (default ./.twd/run) -``` - -And add an example: - -``` - npx twd-cli run --shard 2/4 - npx twd-cli merge .twd/shards -``` - -- [ ] **Step 7: Verify the CLI end to end by hand** - -```bash -node ./bin/twd-cli.js merge -``` - -Expected: prints `Usage: twd-cli merge [--out ]` and exits 1. - -```bash -node ./bin/twd-cli.js merge /tmp/definitely-not-here; echo "exit=$?" -``` - -Expected: prints `No shard reports found in /tmp/definitely-not-here. ...` and `exit=1`. - -```bash -node ./bin/twd-cli.js -``` - -Expected: help text including the `--shard` and `merge` entries. - -- [ ] **Step 8: Run the full suite and commit** - -Run: `npm run test:ci` -Expected: PASS. - -```bash -git add src/mergeCommand.js tests/mergeCommand.test.js bin/twd-cli.js -git commit -m "feat(cli): add the merge command and wire shard flags through bin" -``` - ---- - -### Task 10: End-to-end verification of the CI plumbing - -**Files:** -- Modify: `.github/workflows/e2e.yml` - -**Interfaces:** -- Consumes: the finished CLI from Tasks 1-9. -- Produces: nothing consumed by later tasks. - -Unit tests cannot catch a missing `if: always()`, a wrong artifact path, or a `fail-fast` that cancels siblings. This job is the only place the real plumbing runs. - -- [ ] **Step 1: Add the sharded job** - -Append to `.github/workflows/e2e.yml`, after the existing `e2e` job (same indentation level — two spaces, a sibling of `unit-tests` and `e2e`): - -```yaml - e2e-sharded: - runs-on: ubuntu-latest - - strategy: - # Without this, the first red shard cancels its siblings and the merge job - # sees gaps it cannot distinguish from a crashed shard. - fail-fast: false - matrix: - shard: [1, 2] - - steps: - - name: Checkout repo - uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd # v5 - - - name: Setup Node.js - uses: actions/setup-node@a0853c24544627f65ddf259abe73b1d18a591444 # v5 - with: - node-version: 24 - cache: npm - - - name: Install CLI dependencies - run: npm ci - - - name: Install test-example-app dependencies - working-directory: test-example-app - run: npm install - - - name: Install mock service worker - working-directory: test-example-app - run: npx twd-js init public --save - - - name: Cache Puppeteer browsers - uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4 - with: - path: ~/.cache/puppeteer - key: ${{ runner.os }}-puppeteer-${{ hashFiles('package-lock.json') }} - restore-keys: | - ${{ runner.os }}-puppeteer- - - - name: Install Chrome for Puppeteer - run: npx puppeteer browsers install chrome - - - name: Start dev server - working-directory: test-example-app - run: | - nohup npx vite --host > vite.log 2>&1 & - npx wait-on http://localhost:5173 --timeout 30000 - - - name: Run TWD tests for this shard - working-directory: test-example-app - run: node ../bin/twd-cli.js run --shard ${{ matrix.shard }}/2 - - - name: Upload shard report - # Always: a red shard must still upload, or merge cannot tell "this shard - # failed" from "this shard never ran". - if: always() - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 - with: - name: twd-run-${{ matrix.shard }} - path: test-example-app/.twd/run - if-no-files-found: error - - e2e-merge: - runs-on: ubuntu-latest - needs: [e2e-sharded] - # Runs even though a shard job may have exited 1. Without this the merged - # summary — the point of the exercise — is never printed. - if: ${{ !cancelled() }} - - steps: - - name: Checkout repo - uses: actions/checkout@93cb6efe18208431cddfb8368fd83d5badbf9bfd # v5 - - - name: Setup Node.js - uses: actions/setup-node@a0853c24544627f65ddf259abe73b1d18a591444 # v5 - with: - node-version: 24 - cache: npm - - - name: Install CLI dependencies - run: npm ci - - - name: Download shard reports - uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4 - with: - pattern: twd-run-* - path: test-example-app/.twd/shards - - - name: Merge shard reports - working-directory: test-example-app - run: node ../bin/twd-cli.js merge .twd/shards - - - name: Verify the merged report - working-directory: test-example-app - run: | - if [ ! -f .twd/merged-run.json ]; then - echo "ERROR: merged report not generated" - exit 1 - fi - node -e " - const r = require('./.twd/merged-run.json'); - if (r.shards.length !== 2) { - console.error('ERROR: expected 2 shards, got ' + r.shards.length); - process.exit(1); - } - if (r.tests.length !== r.discovery.totalTests) { - console.error('ERROR: ' + r.tests.length + ' merged tests but ' + - r.discovery.totalTests + ' discovered'); - process.exit(1); - } - console.log('Merged ' + r.tests.length + ' tests from ' + r.shards.length + ' shards'); - " -``` - -- [ ] **Step 2: Verify the action SHAs resolve** - -The `upload-artifact` and `download-artifact` SHAs above must be real v4 tags. Confirm before pushing: - -```bash -gh api repos/actions/upload-artifact/git/ref/tags/v4 --jq .object.sha -gh api repos/actions/download-artifact/git/ref/tags/v4 --jq .object.sha -``` - -Replace the pinned SHAs in the YAML with whatever these print, keeping the `# v4` comment. Every other action in this file is SHA-pinned; these must match that convention. - -- [ ] **Step 3: Validate the YAML parses** - -```bash -node -e " - const fs = require('fs'); - const text = fs.readFileSync('.github/workflows/e2e.yml', 'utf-8'); - if (!text.includes('e2e-sharded') || !text.includes('e2e-merge')) { - throw new Error('jobs missing'); - } - console.log('jobs present'); -" -npx --yes yaml-lint .github/workflows/e2e.yml 2>/dev/null || echo "(yaml-lint unavailable — rely on CI)" -``` - -- [ ] **Step 4: Commit** - -```bash -git add .github/workflows/e2e.yml -git commit -m "ci: verify sharded runs and merge end to end" -``` - -- [ ] **Step 5: Push and confirm the workflow is green** - -```bash -git push -u origin feat/shardable-run-artifacts -gh run watch -``` - -Expected: `unit-tests`, `e2e`, both `e2e-sharded` matrix legs, and `e2e-merge` all pass. If `e2e-merge` reports a missing shard, the upload path or artifact name is wrong — not the merge logic. - ---- - -### Task 11: Documentation and the beta version bump - -**Files:** -- Modify: `README.md` (new section after "CI/CD Integration", before "Contract Validation" at line 270) -- Modify: `CHANGELOG.md` -- Modify: `package.json`, `package-lock.json` - -**Interfaces:** -- Consumes: everything above. -- Produces: a publishable `1.5.0-beta.0`. - -- [ ] **Step 1: Document sharding in the README** - -Insert a `## Sharding across CI jobs` section before `## Contract Validation` (currently line 270): - -````markdown -## Sharding across CI jobs - -A single run walks the whole suite in one browser. `--shard` splits it across -parallel CI jobs instead. - -```bash -npx twd-cli run --shard 2/4 # "I am job 2 of 4" -npx twd-cli merge .twd/shards # join the reports back together -``` - -The `4` is how many jobs you are running, **not** how many tests exist. You never -need to know the test count: each shard boots its own browser, discovers the whole -suite exactly as a normal run does, and keeps every 4th test. Add tests and the -same 4 jobs just split more of them. - -Each shard writes `run.json` and `coverage.json` to `./.twd/run` (change it with -`--report-dir`). `merge` reads the downloaded shard directories, combines the test -results, coverage and contract validation, prints one summary, and exits non-zero -if anything failed anywhere. - -```yaml -jobs: - test: - strategy: - fail-fast: false # or one red shard cancels the rest - matrix: - shard: [1, 2, 3, 4] - steps: - # ...checkout, npm ci, chrome, dev server... - - run: npx twd-cli run --shard ${{ matrix.shard }}/4 - - uses: actions/upload-artifact@v4 - if: always() # a red shard must still upload - with: - name: twd-run-${{ matrix.shard }} - path: .twd/run - - merge: - needs: [test] - if: ${{ !cancelled() }} # runs even though a shard went red - steps: - - uses: actions/checkout@v4 - - uses: actions/setup-node@v4 - with: - node-version: 24 - - run: npm ci - - uses: actions/download-artifact@v4 - with: - pattern: twd-run-* - path: .twd/shards - - run: npx twd-cli merge .twd/shards -``` - -Those three conditions are easy to miss and each one breaks the run: -`fail-fast: false` stops a red shard cancelling its siblings, `if: always()` on -upload keeps a red shard's report, and `if: ${{ !cancelled() }}` on merge lets the -summary print at all. - -### Notes - -- **Coverage.** Each shard writes its own `coverage.json`; `merge` combines them - into `.nyc_output/out.json` — but only when the whole run is green, matching how - a single run behaves. `merge` reports how many shards contributed. -- **Missing shards are an error.** If a shard job dies before uploading, `merge` - refuses and names the gap rather than silently reporting 3 of 4 shards as a - complete green run. -- **Tests must register identically in every job.** Each shard fingerprints the - test list it discovered and `merge` verifies they match. Registering tests - conditionally — behind a feature flag, a date, `Math.random()` — makes the - fingerprints diverge and `merge` will say so. -- **`maxFailures` is per shard.** Four shards at the default of 10 can reach 40 - failures between them before all four bail. -- **`--test` and `--shard` compose:** filters resolve first, then the filtered list - is sharded. As with any filtered run, coverage is skipped. -- **Recording** produces one clip per shard; they are not concatenated. -```` - -- [ ] **Step 2: Add the CHANGELOG entry** - -Prepend to `CHANGELOG.md`, matching the existing `## version (date)` format: - -```markdown -## 1.5.0-beta.0 (2026-08-19) - -* feat(shard): `--shard /` runs one slice of the suite so a run can be split across parallel CI jobs. Each shard discovers the whole suite itself and takes every nth test, so the test count never has to be known in advance -* feat(shard): a sharded run writes `run.json` and `coverage.json` to `./.twd/run` (`--report-dir` to change it) — the first machine-readable output twd-cli has had -* feat(merge): `npx twd-cli merge ` joins shard reports into one report covering test results, coverage and contract validation, prints a single summary with a per-shard breakdown, and owns the exit code -* feat(merge): a missing shard report is an error naming the gap, not a silently incomplete report. Shards also fingerprint the test list they discovered, so shards that saw different test sets refuse to merge -* note: no behavior change without `--shard`. A plain run writes the same files, prints the same output, and exits the same way as 1.4.0 - -Sharding needs three things right in the workflow: `fail-fast: false` on the -matrix, `if: always()` on the shard's artifact upload, and -`if: ${{ !cancelled() }}` on the merge job. See "Sharding across CI jobs" in the -README. - -This is a prerelease, published under the `beta` dist-tag: -`npm install twd-cli@beta`. -``` - -- [ ] **Step 3: Bump the version** - -```bash -npm pkg set version=1.5.0-beta.0 -node -p "require('./package.json').version" -``` - -Expected: `1.5.0-beta.0` - -- [ ] **Step 4: Regenerate the lockfile** - -```bash -npm run lock:linux -``` - -`package-lock.json` carries the version in **two** places — the top-level -`version` and `packages[""].version`. Confirm both moved: - -```bash -node -e " - const lock = require('./package-lock.json'); - const root = lock.packages[''].version; - console.log('top-level:', lock.version, '| packages[\"\"]:', root); - if (lock.version !== '1.5.0-beta.0' || root !== '1.5.0-beta.0') { - throw new Error('lockfile version fields disagree with package.json'); - } -" -``` - -- [ ] **Step 5: Verify the package contents** - -```bash -npm pack --dry-run -``` - -Expected: `bin/`, `src/` (including the five new modules), `README.md`, `LICENSE`. No `tests/`, no `test-example-app/`, no `.twd/`. - -- [ ] **Step 6: Run the full suite one last time** - -```bash -npm run test:ci -``` - -Expected: PASS with no coverage regression on `src/**`. - -- [ ] **Step 7: Commit and push** - -```bash -git add README.md CHANGELOG.md package.json package-lock.json -git commit -m "chore(release): 1.5.0-beta.0" -git push -``` - -- [ ] **Step 8: Hand back for the release** - -The version bump normally happens on `main`, but it lives on this branch by -explicit request so the beta can be tested before merging. Do **not** create the -GitHub Release from this branch. Report to the user that the branch is ready, and -that publishing means: - -1. Merge `feat/shardable-run-artifacts` into `main`. -2. Create a GitHub Release tagged `v1.5.0-beta.0`, **marked as a prerelease**. -3. `publish.yml` sees `prerelease == true` and publishes with `--tag beta`, so - `npm install twd-cli` keeps resolving to 1.4.0. - ---- - -## Verification Checklist - -Run after all tasks are complete. - -- [ ] `npm run test:ci` passes. -- [ ] `node ./bin/twd-cli.js` prints help including `--shard`, `--report-dir` and `merge`. -- [ ] `node ./bin/twd-cli.js merge` exits 1 with the usage message. -- [ ] `node ./bin/twd-cli.js run --shard 5/4` exits 1 with `Invalid --shard`. -- [ ] In `test-example-app` with a dev server running: `node ../bin/twd-cli.js run --shard 1/2` then `--shard 2/2` (moving `.twd/run` to `.twd/shards/a` and `.twd/shards/b` between runs), then `node ../bin/twd-cli.js merge .twd/shards` prints a `Shards: 1 ✓… | 2 ✓…` line and a test count equal to a full unsharded run. -- [ ] Deleting one shard directory and re-running `merge` errors with `Missing shard report(s)`. -- [ ] `node ../bin/twd-cli.js run` with no flags produces byte-identical output to 1.4.0 (`git stash` the branch and compare). -- [ ] CI green on all five jobs. diff --git a/package-lock.json b/package-lock.json index bc71b51..fbe8787 100644 --- a/package-lock.json +++ b/package-lock.json @@ -1,12 +1,12 @@ { "name": "twd-cli", - "version": "1.5.0-beta.0", + "version": "1.5.0", "lockfileVersion": 3, "requires": true, "packages": { "": { "name": "twd-cli", - "version": "1.5.0-beta.0", + "version": "1.5.0", "license": "ISC", "dependencies": { "istanbul-lib-coverage": "^3.2.2", diff --git a/package.json b/package.json index 3649256..abdf87d 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "twd-cli", - "version": "1.5.0-beta.0", + "version": "1.5.0", "description": "CLI tool for running TWD tests with Puppeteer", "type": "module", "main": "src/index.js",