Add: bench env knobs and unified commit/PR message conventions - #819
Conversation
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThe PR rewrites commit, GitHub PR, and iterative fix workflows, and updates golden benchmarking with configurable loop sizes, optional raw timing output, revised L3 reports, and tests covering the new behavior. ChangesWorkflow guidance
Benchmark reporting
Estimated code review effort: 3 (Moderate) | ~25 minutes Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 6
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
.claude/skills/fix-pr/SKILL.md (1)
28-51: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick winUse
gh api graphql’s built-in paginationThe current instruction still asks the caller to rerun the query with a cursor and manually accumulate pages, but a second rerun still writes only the next page to
/tmp/threads.json. Usegh api graphql --paginate --slurpwith the query’s$endCursor: Stringvariable so every page is requested and combined in one output.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.claude/skills/fix-pr/SKILL.md around lines 28 - 51, Update the GraphQL command in the review-thread retrieval instructions to use gh api graphql --paginate --slurp instead of manual cursor reruns and accumulation. Add the pagination variable expected by gh, use it in reviewThreads, and remove the guidance to rerun with -F cursor or manually combine pages; preserve the existing /tmp/threads.json output and subsequent gh pr checks command.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.claude/skills/fix-pr/SKILL.md:
- Around line 104-117: Update the staging instructions in the “Fix and fold into
the PR's commit” section to remove git add -A and stage only the affected file
paths, explicitly including any intended new files. Preserve the existing fetch,
base-reference, and commit-count commands unchanged.
- Around line 141-149: Update the “Resolve comment threads” instructions to use
{owner} and {repo} placeholders in the gh api repository endpoint instead of
literal :owner and :repo segments. Ensure the surrounding guidance provides
enough repository context for GitHub CLI to resolve those placeholders, while
preserving the existing reply and resolve workflow.
In @.claude/skills/git-commit/SKILL.md:
- Around line 68-75: The commit-body guidance in the staged-diff verification
section conflicts with the Perf body requirement. Update the rule to require a
reproducible benchmark command and configuration for measured before/after
values, or explicitly permit external measurements only when their reproduction
details are included; apply the same clarification to the corresponding guidance
around the Perf format.
In @.claude/skills/github-pr/SKILL.md:
- Around line 22-44: Update the effectively-main branch flow in the github-pr
instructions to define commit subject generation before branch creation. Derive
the commit type and slug from the staged diff, then use that subject to select
the branch prefix and construct the branch name before invoking /git-commit;
preserve the existing naming rules and subsequent commit/PR actions.
- Around line 55-63: Require explicit human confirmation immediately before the
force-push step in both workflows: update .claude/skills/github-pr/SKILL.md
lines 55-63 around the rebase/push instructions, and
.claude/skills/fix-pr/SKILL.md lines 134-139 around its rewritten-commit push,
while leaving non-force pushes unchanged.
In `@tests/golden/test_runner.py`:
- Around line 1477-1481: Update the _FakeStats test double so mutable list
attributes are created per instance rather than shared at class scope,
addressing RUF012. Initialize the relevant invocation, dispatch, and wall-time
collections in _FakeStats.__init__ while preserving their existing defaults and
behavior.
---
Outside diff comments:
In @.claude/skills/fix-pr/SKILL.md:
- Around line 28-51: Update the GraphQL command in the review-thread retrieval
instructions to use gh api graphql --paginate --slurp instead of manual cursor
reruns and accumulation. Add the pagination variable expected by gh, use it in
reviewThreads, and remove the guidance to rerun with -F cursor or manually
combine pages; preserve the existing /tmp/threads.json output and subsequent gh
pr checks command.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: a339ca84-73ab-4193-a205-086071bb36cf
📒 Files selected for processing (6)
.claude/skills/fix-pr/SKILL.md.claude/skills/git-commit/SKILL.md.claude/skills/github-pr/SKILL.mdgolden/runner.pytests/golden/conftest.pytests/golden/test_runner.py
- Commit types are now Add/Fix/Perf/Refactor/Docs/CI/Chore. 'Update' is dropped and the conventional-commit `feat(scope):` form is rejected. github-pr maps each type to a branch prefix and reuses the commit subject verbatim as the PR title. - Commit and PR bodies describe the diff, not the session that produced it: no lint / test-run logs, commands invoked, or debugging path. `## Summary` is the only PR section, and an issue link is a bare trailing `Fixes #N`, so no empty `## Related Issues` heading is left behind. - A `Perf:` body must state the measured before -> after and the configuration it was measured on. - fix-pr folds fixes into the PR's existing commit (amend, or soft-reset squash when more than one commit is ahead of base) instead of appending a `fix(pr):` commit, and evolves that commit's message rather than freezing or replacing it. - fix-pr swaps the `sleep 600` wait for `gh pr checks --watch` and adds the per-job log fetch, which works while a run is still in progress. - git-commit forbids `--no-verify` and `SKIP=`, and covers kernels under `models/` alongside `examples/`. Drop the Task Tracking, Checklist, Common Issues, Best Practices, and workflow-diagram sections that restated the steps: 550 -> 395 lines.
- PYPTO_BENCH_RAW prints every measured dispatch's Effective sample per rank (L2, L3, and the L3 flatten fallback where the per-round summaries are least trustworthy); off by default - PYPTO_BENCH_ROUNDS / PYPTO_BENCH_WARMUP override the 100 / 5 loop sizes, which no longer fit a long prefill or a multi-card L3 run; a malformed value warns and falls back instead of failing the run - The resident L3 path clamps warmup to >= 1 via _resident_loop_sizes: it spends warmup[0] on the validation dispatch, so warmup=0 would emit rounds+1 dispatches and lose per-round segmentation - Drop device_wall from every report (headline, per-rank, L3 detail); Effective is the metric consumed downstream - Unit tests for the knobs plus a guard that exactly one line matches daily CI's `effective_us .*mean=` collector - conftest clears the PYPTO_BENCH* vars: an exported PYPTO_BENCH sent run() down the benchmark path the stub pypto cannot serve, failing 22 unrelated tests Daily CI sets none of the new vars, so its 100 / 5 perf baseline and the collected number are unchanged.
- performance-tuning.md gains a "Measuring" section ahead of Part 1: how to enable the timed loop, what Effective means, the `mean=` field daily CI collects, the real-device / SIMPLER_PROFILING requirements, how to read the multi-card per-rank and fallback_flattened output, and the four PYPTO_BENCH* knobs - compile-runtime-workflow.md gains phase 4b — the loop runs after the correctness dispatch and before validation, and was undocumented
- Remove docs/investigations/2026-07-qwen3-fused-attention-sync-barrier.md - cce-extern-kernel-guide.md keeps the conclusion (the C220 MTE3/TSTORE to MTE2 path was validated over 40 layers without an extra DDR barrier) and links pypto-lib#796 in place of the deleted file
4149a74 to
2d68cfb
Compare
The qwen3-14b decode kernel was renamed to decode_fwd.py in hw-native-sys#796, leaving every doc example pointing at a file that no longer exists. - Repoint the path in README.md, AGENTS.md, .claude/CLAUDE.md, performance-tuning.md, tools/README.md, and the two profiling helpers' usage docstrings - `--enable-pmu` is not on that kernel: the PMU example now uses deepseek/v4-flash/decode_sparse_attn.py and notes the runtime_cfg={"enable_pmu": 2} route for kernels without the flag - `--export-kernel-insight` no longer exists on any kernel: the insight examples now call tools/export_all_kernel_insight.py directly - Swap the deprecated `--runtime-profiling` alias for `--enable-l2-swimlane` in both helper docstrings
Summary
Benchmark knobs in
golden/runner.py:PYPTO_BENCH_RAWprints every measured dispatch's Effective sample per rank — for L2, L3, and the L3 flatten fallback where the per-round summaries are least trustworthy; off by defaultPYPTO_BENCH_ROUNDS/PYPTO_BENCH_WARMUPoverride the hardcoded 100 / 5 loop sizes, which do not fit a long prefill or a multi-card L3 run; a malformed value warns and falls back instead of failing the run_resident_loop_sizes: it spends warmup[0] on the validation dispatch, so warmup=0 would emit rounds+1 dispatches and lose per-round segmentationdevice_wallfrom every report (headline, per-rank, L3 detail); Effective is the metric consumed downstream. Daily CI sets none of the new vars, so its 100 / 5 baseline and theeffective_us .*mean=number it collects are unchangedtests/golden/conftest.pyclears thePYPTO_BENCH*vars, which previously failed 22 unrelated tests when exportedBenchmark documentation:
docs/performance-tuning.mdgains a "Measuring" section ahead of Part 1: enabling the timed loop, what Effective is, themean=field daily CI collects, the real-device /SIMPLER_PROFILINGrequirements, reading the multi-card per-rank andfallback_flattenedoutput, and the fourPYPTO_BENCH*knobsdocs/compile-runtime-workflow.mdgains phase 4b — the loop runs after the correctness dispatch and before validation, and was undocumentedCommit / PR conventions in
.claude/skills/:git-commit,github-prandfix-prrewritten against one shared convention: the seven commit types, a body derived only from the staged diff orBASE_REF..HEAD,## Summaryas the sole PR section, and no AI co-author footersgit-commitgains the type table with per-type usage, the rule that aPerf:body states measured before -> after plus the configuration, and an explicit no-bypass rule for pre-commit hooksgithub-prreplaces the prose walk-through with the state table (effectively-on-main x uncommitted), the commit-type-to-branch-prefix table, and--force-with-leaseafter a rebasefix-prdrops the ASCII flow diagram for numbered steps with a 5-iteration capDocumentation cleanup:
docs/investigations/2026-07-qwen3-fused-attention-sync-barrier.md;cce-extern-kernel-guide.mdkeeps the conclusion (the C220 MTE3/TSTORE-to-MTE2 path was validated over 40 layers without an extra DDR barrier) and cites perf(qwen3-14b decode): fold RoPE into the CANN FAI attention extern #796 in place of the deleted file