Skip to content

Add: bench env knobs and unified commit/PR message conventions - #819

Merged
zhangqi-chen merged 5 commits into
hw-native-sys:mainfrom
zhangqi-chen:docs/unify-commit-and-pr-message-conventions
Jul 23, 2026
Merged

Add: bench env knobs and unified commit/PR message conventions#819
zhangqi-chen merged 5 commits into
hw-native-sys:mainfrom
zhangqi-chen:docs/unify-commit-and-pr-message-conventions

Conversation

@zhangqi-chen

@zhangqi-chen zhangqi-chen commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

Summary

Benchmark knobs in golden/runner.py:

  • PYPTO_BENCH_RAW prints every measured dispatch's Effective sample per rank — for L2, L3, and the L3 flatten fallback where the per-round summaries are least trustworthy; off by default
  • PYPTO_BENCH_ROUNDS / PYPTO_BENCH_WARMUP override the hardcoded 100 / 5 loop sizes, which do not fit a long prefill or a multi-card L3 run; a malformed value warns and falls back instead of failing the run
  • The resident L3 path clamps warmup to >= 1 via _resident_loop_sizes: it spends warmup[0] on the validation dispatch, so warmup=0 would emit rounds+1 dispatches and lose per-round segmentation
  • Drop device_wall from every report (headline, per-rank, L3 detail); Effective is the metric consumed downstream. Daily CI sets none of the new vars, so its 100 / 5 baseline and the effective_us .*mean= number it collects are unchanged
  • Unit tests for the knobs plus a guard that exactly one report line matches daily CI's collector; tests/golden/conftest.py clears the PYPTO_BENCH* vars, which previously failed 22 unrelated tests when exported

Benchmark documentation:

  • docs/performance-tuning.md gains a "Measuring" section ahead of Part 1: enabling the timed loop, what Effective is, the mean= field daily CI collects, the real-device / SIMPLER_PROFILING requirements, reading the multi-card per-rank and fallback_flattened output, and the four PYPTO_BENCH* knobs
  • docs/compile-runtime-workflow.md gains phase 4b — the loop runs after the correctness dispatch and before validation, and was undocumented

Commit / PR conventions in .claude/skills/:

  • git-commit, github-pr and fix-pr rewritten against one shared convention: the seven commit types, a body derived only from the staged diff or BASE_REF..HEAD, ## Summary as the sole PR section, and no AI co-author footers
  • git-commit gains the type table with per-type usage, the rule that a Perf: body states measured before -> after plus the configuration, and an explicit no-bypass rule for pre-commit hooks
  • github-pr replaces the prose walk-through with the state table (effectively-on-main x uncommitted), the commit-type-to-branch-prefix table, and --force-with-lease after a rebase
  • fix-pr drops the ASCII flow diagram for numbered steps with a 5-iteration cap

Documentation cleanup:

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@coderabbitai

coderabbitai Bot commented Jul 23, 2026

Copy link
Copy Markdown

Review Change Stack

Important

Review skipped

Auto incremental reviews are disabled on this repository.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: a555de53-b1e8-43fd-8d66-dc30177abefd

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The PR rewrites commit, GitHub PR, and iterative fix workflows, and updates golden benchmarking with configurable loop sizes, optional raw timing output, revised L3 reports, and tests covering the new behavior.

Changes

Workflow guidance

Layer / File(s) Summary
Commit and PR creation workflow
.claude/skills/git-commit/SKILL.md, .claude/skills/github-pr/SKILL.md
Commit checks, branch naming, rebasing, authentication, and PR title/body rules are reorganized into explicit numbered workflows.
Iterative PR fixing
.claude/skills/fix-pr/SKILL.md
PR input handling, review-thread and CI detection, issue classification, consent, commit folding, thread resolution, polling, and error handling are rewritten for an iterative five-pass workflow.

Benchmark reporting

Layer / File(s) Summary
Benchmark sizing and reporting
golden/runner.py
Benchmark rounds and warmup values can be configured through environment variables; raw per-dispatch samples are optionally reported, and L3 output omits device-wall metrics.
Benchmark path integration
golden/runner.py
L2, non-resident L3, resident L3, and JIT paths use dynamic loop sizes and invoke raw reporting.
Benchmark validation
tests/golden/conftest.py, tests/golden/test_runner.py
Tests isolate benchmark environment variables and cover loop-size defaults, overrides, invalid values, resident warmup behavior, and report formatting.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Possibly related PRs

Poem

A rabbit reviews the loops with care,
Raw timing samples hop through the air.
PR threads resolve, commits neatly fold,
Warmups adjust as benchmarks are told.
“Green checks!” I cheer, with a twitch of my nose—
New guidance and reports in orderly rows.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title matches the main changes: benchmark env knobs and shared commit/PR message conventions.
Description check ✅ Passed The description is directly related and summarizes the benchmark and workflow documentation changes.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@zhangqi-chen zhangqi-chen changed the title Add: PYPTO_BENCH_RAW / _ROUNDS / _WARMUP benchmark knobs Add: bench env knobs and unified commit/PR message conventions Jul 23, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
.claude/skills/fix-pr/SKILL.md (1)

28-51: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Use gh api graphql’s built-in pagination

The current instruction still asks the caller to rerun the query with a cursor and manually accumulate pages, but a second rerun still writes only the next page to /tmp/threads.json. Use gh api graphql --paginate --slurp with the query’s $endCursor: String variable so every page is requested and combined in one output.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.claude/skills/fix-pr/SKILL.md around lines 28 - 51, Update the GraphQL
command in the review-thread retrieval instructions to use gh api graphql
--paginate --slurp instead of manual cursor reruns and accumulation. Add the
pagination variable expected by gh, use it in reviewThreads, and remove the
guidance to rerun with -F cursor or manually combine pages; preserve the
existing /tmp/threads.json output and subsequent gh pr checks command.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.claude/skills/fix-pr/SKILL.md:
- Around line 104-117: Update the staging instructions in the “Fix and fold into
the PR's commit” section to remove git add -A and stage only the affected file
paths, explicitly including any intended new files. Preserve the existing fetch,
base-reference, and commit-count commands unchanged.
- Around line 141-149: Update the “Resolve comment threads” instructions to use
{owner} and {repo} placeholders in the gh api repository endpoint instead of
literal :owner and :repo segments. Ensure the surrounding guidance provides
enough repository context for GitHub CLI to resolve those placeholders, while
preserving the existing reply and resolve workflow.

In @.claude/skills/git-commit/SKILL.md:
- Around line 68-75: The commit-body guidance in the staged-diff verification
section conflicts with the Perf body requirement. Update the rule to require a
reproducible benchmark command and configuration for measured before/after
values, or explicitly permit external measurements only when their reproduction
details are included; apply the same clarification to the corresponding guidance
around the Perf format.

In @.claude/skills/github-pr/SKILL.md:
- Around line 22-44: Update the effectively-main branch flow in the github-pr
instructions to define commit subject generation before branch creation. Derive
the commit type and slug from the staged diff, then use that subject to select
the branch prefix and construct the branch name before invoking /git-commit;
preserve the existing naming rules and subsequent commit/PR actions.
- Around line 55-63: Require explicit human confirmation immediately before the
force-push step in both workflows: update .claude/skills/github-pr/SKILL.md
lines 55-63 around the rebase/push instructions, and
.claude/skills/fix-pr/SKILL.md lines 134-139 around its rewritten-commit push,
while leaving non-force pushes unchanged.

In `@tests/golden/test_runner.py`:
- Around line 1477-1481: Update the _FakeStats test double so mutable list
attributes are created per instance rather than shared at class scope,
addressing RUF012. Initialize the relevant invocation, dispatch, and wall-time
collections in _FakeStats.__init__ while preserving their existing defaults and
behavior.

---

Outside diff comments:
In @.claude/skills/fix-pr/SKILL.md:
- Around line 28-51: Update the GraphQL command in the review-thread retrieval
instructions to use gh api graphql --paginate --slurp instead of manual cursor
reruns and accumulation. Add the pagination variable expected by gh, use it in
reviewThreads, and remove the guidance to rerun with -F cursor or manually
combine pages; preserve the existing /tmp/threads.json output and subsequent gh
pr checks command.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: a339ca84-73ab-4193-a205-086071bb36cf

📥 Commits

Reviewing files that changed from the base of the PR and between 72ee6a1 and f9fb940.

📒 Files selected for processing (6)
  • .claude/skills/fix-pr/SKILL.md
  • .claude/skills/git-commit/SKILL.md
  • .claude/skills/github-pr/SKILL.md
  • golden/runner.py
  • tests/golden/conftest.py
  • tests/golden/test_runner.py

Comment thread .claude/skills/fix-pr/SKILL.md
Comment thread .claude/skills/fix-pr/SKILL.md
Comment thread .claude/skills/git-commit/SKILL.md
Comment thread .claude/skills/github-pr/SKILL.md
Comment thread .claude/skills/github-pr/SKILL.md
Comment thread tests/golden/test_runner.py Outdated
- Commit types are now Add/Fix/Perf/Refactor/Docs/CI/Chore. 'Update' is
  dropped and the conventional-commit `feat(scope):` form is rejected.
  github-pr maps each type to a branch prefix and reuses the commit
  subject verbatim as the PR title.
- Commit and PR bodies describe the diff, not the session that produced
  it: no lint / test-run logs, commands invoked, or debugging path.
  `## Summary` is the only PR section, and an issue link is a bare
  trailing `Fixes #N`, so no empty `## Related Issues` heading is left
  behind.
- A `Perf:` body must state the measured before -> after and the
  configuration it was measured on.
- fix-pr folds fixes into the PR's existing commit (amend, or soft-reset
  squash when more than one commit is ahead of base) instead of
  appending a `fix(pr):` commit, and evolves that commit's message
  rather than freezing or replacing it.
- fix-pr swaps the `sleep 600` wait for `gh pr checks --watch` and adds
  the per-job log fetch, which works while a run is still in progress.
- git-commit forbids `--no-verify` and `SKIP=`, and covers kernels under
  `models/` alongside `examples/`.

Drop the Task Tracking, Checklist, Common Issues, Best Practices, and
workflow-diagram sections that restated the steps: 550 -> 395 lines.
- PYPTO_BENCH_RAW prints every measured dispatch's Effective sample
  per rank (L2, L3, and the L3 flatten fallback where the per-round
  summaries are least trustworthy); off by default
- PYPTO_BENCH_ROUNDS / PYPTO_BENCH_WARMUP override the 100 / 5 loop
  sizes, which no longer fit a long prefill or a multi-card L3 run; a
  malformed value warns and falls back instead of failing the run
- The resident L3 path clamps warmup to >= 1 via _resident_loop_sizes:
  it spends warmup[0] on the validation dispatch, so warmup=0 would
  emit rounds+1 dispatches and lose per-round segmentation
- Drop device_wall from every report (headline, per-rank, L3 detail);
  Effective is the metric consumed downstream
- Unit tests for the knobs plus a guard that exactly one line matches
  daily CI's `effective_us .*mean=` collector
- conftest clears the PYPTO_BENCH* vars: an exported PYPTO_BENCH sent
  run() down the benchmark path the stub pypto cannot serve, failing
  22 unrelated tests

Daily CI sets none of the new vars, so its 100 / 5 perf baseline and
the collected number are unchanged.
- performance-tuning.md gains a "Measuring" section ahead of Part 1: how
  to enable the timed loop, what Effective means, the `mean=` field daily
  CI collects, the real-device / SIMPLER_PROFILING requirements, how to
  read the multi-card per-rank and fallback_flattened output, and the
  four PYPTO_BENCH* knobs
- compile-runtime-workflow.md gains phase 4b — the loop runs after the
  correctness dispatch and before validation, and was undocumented
- Remove docs/investigations/2026-07-qwen3-fused-attention-sync-barrier.md
- cce-extern-kernel-guide.md keeps the conclusion (the C220 MTE3/TSTORE
  to MTE2 path was validated over 40 layers without an extra DDR
  barrier) and links pypto-lib#796 in place of the deleted file
@zhangqi-chen
zhangqi-chen force-pushed the docs/unify-commit-and-pr-message-conventions branch from 4149a74 to 2d68cfb Compare July 23, 2026 02:16
The qwen3-14b decode kernel was renamed to decode_fwd.py in hw-native-sys#796, leaving
every doc example pointing at a file that no longer exists.

- Repoint the path in README.md, AGENTS.md, .claude/CLAUDE.md,
  performance-tuning.md, tools/README.md, and the two profiling helpers'
  usage docstrings
- `--enable-pmu` is not on that kernel: the PMU example now uses
  deepseek/v4-flash/decode_sparse_attn.py and notes the
  runtime_cfg={"enable_pmu": 2} route for kernels without the flag
- `--export-kernel-insight` no longer exists on any kernel: the insight
  examples now call tools/export_all_kernel_insight.py directly
- Swap the deprecated `--runtime-profiling` alias for
  `--enable-l2-swimlane` in both helper docstrings
@zhangqi-chen
zhangqi-chen merged commit dcdd061 into hw-native-sys:main Jul 23, 2026
9 checks passed
@zhangqi-chen
zhangqi-chen deleted the docs/unify-commit-and-pr-message-conventions branch July 23, 2026 02:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant