Include static hermes results in benchmark report#328
Merged
Conversation
- bench_and_report.ts now runs both dynamic and static benchmarks, writing bench_dynamic_result.json and bench_static_result.json, then merges them into bench_result.json (static entries suffixed with " (static)"); the merged file is the only one that carries a timestamp. - report.ts rounds the per-group millisecond sums in the collapsible summary to integers, and appends a +/- percentage delta to the summary title when comparing exactly two inputs. - .gitignore picks up the two new intermediate JSON files. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
vmoroz
approved these changes
May 12, 2026
Without these the static benchmark phase invokes hermes.exe as if it were shermes.exe and bails on the first compile with "Unknown command line argument '-o'". The workflow now builds the shermes target and passes a separate --static-binary so bench_and_report.ts feeds the dynamic and static phases their respective binaries. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
shermes spawns clang to link a native executable, and clang on Windows needs LIB/INCLUDE/PATH from vcvarsall to find link.exe and the MSVC runtime libs. build.js sets those inside its own process for the build step, but the env doesn't survive into the subsequent bash step where the benchmarks run. Without this, the static phase fails on the first benchmark with LNK1181. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous attempt used ilammy/msvc-dev-cmd@v1, which microsoft/ hermes-windows does not allow (the workflow exited with startup_failure in 0s). Replace it with an inline cmd step that discovers the VS installation through vswhere, sources vcvarsall.bat, and exports the relevant env vars (PATH, LIB, INCLUDE, LIBPATH) to GITHUB_ENV so the subsequent bash bench step inherits them. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The previous attempt built only the 'shermes' target, which produces shermes.exe but not the runtime libraries (hermesvm, shermes_console, their static counterparts) that the generated native executables link against. The static phase consequently failed on the first benchmark with LNK1181. tools/shermes/CMakeLists.txt provides 'shermes-dep' as a meta-target that pulls in shermes plus all the libraries needed at link time, so switch the build to that. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Benchmark ResultsTotal benchmarks: 204 Inputs: baseline/baseline.json, bench_result.json v8 (3354ms, 6247ms) +86.2%
test-suites (177934ms, 312812ms) +75.8%
micros (32446ms, 61591ms) +89.8%
jit-benches (6261ms, 8583ms) +37.1%
many-subclasses (48717ms, 73105ms) +50.1%
map-objects (2027ms, 3616ms) +78.4%
map-strings (2415ms, 4323ms) +79.0%
nbody (2478ms, 3427ms) +38.3%
string-switch (1335ms, 6548ms) +390.3%
raytracer (3319ms, 5604ms) +68.9%
MiniReact (17368ms, 30006ms) +72.8%
widgets (7554ms, 12874ms) +70.4%
|
vmoroz
approved these changes
May 12, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
With a few formatting
Test Plan
It also tests about report generation when local benchmark data has extra items than baseline.
Microsoft Reviewers: Open in CodeFlow