benchcheck: parse go test -json instead of scanning text

[michelle-clayton-work]

Replace the prefix-matching failure scanner with structured test2json
parsing, so failure detection no longer depends on matching go test
text output line-by-line.

What changed

• New testjson.go — parseTestJSON reads go test -json output and:

  • detects build errors, failed tests, and crashes structurally from
    fail events (flushing the failing package/test's buffered output for
    context, dropping the === RUN/=== PAUSE/=== CONT framing lines),
    instead of pattern-matching # , --- FAIL, FAIL\t, SIGSEGV:, and
    panic: prefixes;
  • reconstructs the plain benchmark text from the output events, which
    both benchfmt (regression parsing) and benchstat consume;
  • tolerates stray non-JSON lines.

• check.go — now takes base.json/head.json, parses each once for
  both failures and benchmarks, and adds -o-base-text/-o-head-text flags
  that write the reconstructed plain text for benchstat. Removed the old
  extractFailures, isCrashLine, appendFailuresFromFile, and
  parseBenchmarksFromFile helpers.

• benchmark.yml — the bench steps now emit go test -json to
  base.json/head.json. The "Compare and check" step runs benchcheck
  first (capturing its exit code) so it can produce the reconstructed
  base.txt/head.txt, then runs benchstat on them, then propagates the
  check exit code — so benchstat.txt is still produced even when a
  regression or failure makes check exit non-zero.

Why

The text scanner was fragile: it relied on the exact shape of go test
output and on heuristics (e.g. cutting a crash trace at the first blank
line). Parsing the structured test2json stream makes failure detection
robust and lets us attribute output to the specific package or test that
failed.

Testing

• gofmt, go vet, go test ./cmd/benchcheck/ all clean; the text-scanner
  unit tests were replaced with parseTestJSON tests covering test
  failures, build failures, crashes, passing benchmark reconstruction, and
  non-JSON tolerance.
• End-to-end smoke test against crafted test2json inputs: reconstructed
  benchmark text is benchstat-ready, failures.txt captures the failing
  test and package summary, and status.json reflects the failure.

[Copilot]

Pull request overview

This PR makes benchcheck consume go test -json (test2json) output instead of scanning plain text, enabling structured failure detection and reconstruction of benchmark text for regression analysis and benchstat.

Changes:

• Added parseTestJSON to parse test2json streams, extract failures from fail events, and reconstruct benchstat/benchfmt-compatible plain text.
• Updated benchcheck check to accept base.json/head.json, parse each input once, and optionally write reconstructed base.txt/head.txt.
• Updated the benchmark GitHub Actions workflow to emit base.json/head.json and run benchcheck before benchstat while preserving the check exit code.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
cmd/benchcheck/testjson.go	Introduces test2json parsing, benchmark text reconstruction, and structured failure extraction.
cmd/benchcheck/check.go	Switches check to parse .json inputs and adds flags to emit reconstructed plain-text outputs for benchstat.
cmd/benchcheck/main.go	Updates CLI docs to reflect .json inputs for the check subcommand.
cmd/benchcheck/main_test.go	Replaces text-scanner tests with parseTestJSON coverage for failures, crashes, and non-JSON tolerance.
.github/workflows/benchmark.yml	Updates workflow to write test2json outputs and use benchcheck to produce base.txt/head.txt before running benchstat.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[dagood]

Looks good, a couple alternatives for a structural nitpick.

[dagood]

Would it make more sense to run benchstat as part of benchcheck, avoiding some shuffling of data through bash (and through files)?

[michelle-clayton-work]

Took another look into this and the pinned x/perf only exposes the deprecated benchstat library whose output differs from the cmd/benchstat CLI we use. Reproducing the current format in-process would mean coupling to benchstat's internals and I feel like that would be a lot of work for a smaller result so maybe I would lean towards keeping it as a CLI call

[dagood]

I'm not talking about it being in-process necessarily--the benchcheck tool could use os/exec. Maybe it's a wash, with the changes needed to do that. 🤷

(If nobody will be attempting this sequence of commands locally, we also don't have that reason to try to make benchcheck handle more for them.)

[michelle-clayton-work]

updated the reference here to test the changes

• test: benchcheck variance-detection [do not merge] go-crypto-winnative#187

[dagood]

I'm not talking about it being in-process necessarily--the benchcheck tool could use os/exec. Maybe it's a wash, with the changes needed to do that. 🤷

(If nobody will be attempting this sequence of commands locally, we also don't have that reason to try to make benchcheck handle more for them.)