test(headless): run ordinary CLI command semantics in process - #2479
Closed
UncertaintyDeterminesYou4ndMe wants to merge 1 commit into
Closed
Conversation
Part of apache#2387. The Headless CLI, contamination-scan, and harness A/B suites started a Node subprocess per assertion for scenarios that only exercise argument validation, task-run business logic, and report verdicts — semantics the exported entry points already expose. - cli.test.ts: 10 of 11 tests now run through mapLegacyMakaHeadlessArgs + runMakaEvalCli with stdout/stderr captured at the process-stream seam and env overrides applied and restored around the call. The non-Headless-root test keeps the real bin route as the representative wiring contract (real exit code, stack-free stderr). - contamination-scan-cli.test.ts: 13 of 14 tests call the script's exported main(argv), mirroring the executable footer exactly (thrown error -> stderr + exit 2). The no-argument rejection keeps the real subprocess as representative coverage of that footer itself, including its realpath main-module guard. - harness-ab-cli.test.ts: the three pure pre-launch validation tests call the exported main() with a scoped env; every remaining subprocess in the file exercises the detached launcher, its crash-time journal, or the credentials-after-validation order — contracts that need a real process. - New shared helper withCapturedProcessIo swaps and restores the process-wide stream writers; safe for the sequential node:test runs these files use. runtime-policy-ab-cli.test.ts is intentionally untouched: its single test is the representative subprocess for run-runtime-policy-ab.mjs, whose main() is not exported, and adding an export to shave 0.6s is not warranted. Timing (node --test, local, warm build): cli.test.js 21.8s -> 13.1s (spawns 20 -> 2) contamination-scan-cli.test.js 1.05s -> 0.63s (spawns 14 -> 1) harness-ab-cli.test.js 7.81s -> 5.86s (spawns 9 -> 6) 69/69 tests pass across 3 consecutive rounds.
UncertaintyDeterminesYou4ndMe
force-pushed
the
perf/2387-headless-in-process
branch
from
August 8, 2026 04:49
b5b4fab to
6bfc7b3
Compare
Contributor
Author
|
GitHub never generated check suites for this branch (0 runs across create/reopen/force-push — a platform-side event delivery failure, not a CI config issue). Superseding with an identical PR from a fresh branch so CI can run. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #2387 (Headless workspace; the CLI workspace landed separately as #2476 per the issue's separate-PR requirement).
Problem
Three Headless suites started a Node subprocess per assertion for scenarios that only exercise argument validation, task-run business logic, and report verdicts.
cli.test.tswas the sharpest case: 11 tests, 20 subprocess launches, 21.8s — each launch paying the full module-graph load of the headless CLI to test semantics thatrunMakaEvalClialready exposes as an exportedPromise<number>router.Change
No production changes. Every conversion goes through an already-exported entry:
cli.test.ts— 10 of 11 tests run throughmapLegacyMakaHeadlessArgs+runMakaEvalCli, the same mapping and router the bin runs, with stdout/stderr captured at the process-stream seam and env overrides applied/restored around the call. The non-Headless-root test keeps the real bin route as the representative wiring contract (real exit code, stack-free stderr hygiene).contamination-scan-cli.test.ts— 13 of 14 tests call the script's exportedmain(argv), with the executable footer's contract mirrored exactly (thrown error → stderr + exit 2, verdict codes returned). The no-argument rejection keeps the real subprocess as representative coverage of the footer itself, including its realpath main-module guard. The--markdowntest gains an explicitassert.equal(code, 0)that was previously implicit inexecFileAsyncnot rejecting.harness-ab-cli.test.ts— the three pure pre-launch validation tests (unsupported composition, superseded 30-task pilot, 89-task profile fingerprint mismatch) call the exportedmain()with a scoped env. Every remainingexecFileAsyncin the file exercises the detached launcher, its crash-time journal, the run lock, or the credentials-only-after-validation order — contracts that need a real process and stay on one.withCapturedProcessIoswaps and restores the process-wide stream writers; safe for the sequentialnode:testexecution these files use.Deliberately not converted
runtime-policy-ab-cli.test.ts: its single test is the representative subprocess forrun-runtime-policy-ab.mjs, whosemain()is not exported — adding an export to shave 0.6s is a production touch the saving does not justify.pi-cli-json-transport.test.tsalready uses an injected fake child;task-agent-controller/harbor-cellsubprocesses are the behavior under test.Retained process contracts
Timing
node --test, same machine, warm build:The residual 13s in
cli.test.jsis dominated by one heavy-task journey (7.4s) whose cost is task-run business logic, not startup tax — visible now that the tax is gone.69/69 tests pass across 3 consecutive rounds; typecheck, biome, and knip are clean (knip delta against baseline:
run-contamination-scan.mjsmoves from unused to used).cc @Astro-Han