Skip to content

test: publish task-aware benchmark medians - #120

Merged
yashranaway merged 2 commits into
mainfrom
test/benchmark-json-medians
Aug 12, 2026
Merged

test: publish task-aware benchmark medians#120
yashranaway merged 2 commits into
mainfrom
test/benchmark-json-medians

Conversation

@SarthakWade

@SarthakWade SarthakWade commented Aug 12, 2026

Copy link
Copy Markdown
Collaborator

Closes #38

Summary

  • add task-aware inspect --context actions --task calls to the cold and warm Headless benchmark workflows
  • collect repeat runs as strict NDJSON and publish a versioned JSON document with every raw sample and per-metric medians
  • refresh the canonical Linux ARM64 snapshot from five fresh containers per case
  • document the mixed measured result: Headless warm has the smallest workflow surface and lowest median CPU, while Puppeteer is fastest and has the lowest median peak memory
  • add deterministic regression coverage and mark backlog item D5 complete

Result artifact

packages/benchmark-results/results.json records:

  • 20 raw samples across four workflows
  • median wall time, CPU time, peak memory, artifact bytes, workflow bytes, and estimated tokens
  • generation time, Linux container architecture, repeat count, aggregation method, generator, method document, and task-aware status

The default benchmark output is canonical; a second CLI argument writes an independent comparison artifact.

Hardening

  • reject zero, non-numeric, and excessive repeat counts before Docker work begins
  • reject malformed JSON, unknown cases, unexpected fields, unsafe numeric metrics, incomplete case sets, and inconsistent source-size metrics
  • write results atomically through a private temporary file, publish as mode 0644, and preserve an existing result on validation failure
  • use a run-specific Docker image tag and remove it on success, interruption, or failure
  • run aggregation inside the benchmark image so Docker remains the only host dependency
  • retain all raw samples so outliers remain visible instead of being hidden by aggregation

Verification

  • ./apps/headless/benchmark.sh 5 completed 20 fresh containers and generated the committed snapshot
  • ./apps/headless/benchmark.sh 1 /private/tmp/headless-benchmark-smoke.json verified custom output, permissions, and Docker image cleanup
  • a second four-container smoke run verified containerized aggregation with no host Node dependency
  • shellcheck apps/headless/benchmark.sh apps/headless/Tests/Benchmarks/headless.sh apps/headless/Tests/Benchmarks/headless_warm.sh
  • pnpm test
  • pnpm test:runtime
  • pnpm --filter @headless/web lint
  • pnpm --filter @headless/web build
  • HEADLESS_VERSION=0.0.38 ./apps/headless/build.sh
  • pnpm test:e2e:mac
  • pnpm test:e2e:linux
  • git diff --check

@SarthakWade
SarthakWade marked this pull request as ready for review August 12, 2026 14:31
@yashranaway
yashranaway merged commit 9d05a1b into main Aug 12, 2026
7 checks passed
@yashranaway
yashranaway deleted the test/benchmark-json-medians branch August 12, 2026 15:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

D5 — Benchmark refresh discipline: emit JSON results and re-run with --task

2 participants