docs(benchmarking): reflect bench_gpu.py + bench_rf.py added in PR #253

[claude]

Summary

Follow-up to merged #253 (feat(benchmarks): GPU benchmark for Beta-VAE train step + encoder inference).

docs/BENCHMARKING.md was framed as CPU-only and enumerated benchmarks/ scripts in two places — both went stale when #253 landed benchmarks/bench_gpu.py and benchmarks/bench_rf.py. This PR applies exactly the five contiguous edits called out in #266 to bring it back in line with benchmarks/README.md; nothing else in the tree needs reconciliation.

• TL;DR bullet — rewritten to cover the CPU/GPU mix instead of "hot CPU kernels".
• Section heading + intro — renamed Micro-benchmarks (benchmarks/) to Standalone benchmarks (benchmarks/) and rewrote the opener (CPU micro-benchmarks + bench_gpu real-GPU profiler; still uncollected by pytest; still writes JSON to benchmarks/results/). Also refreshed the matching phrase in the top-of-file intro paragraph. No file-internal anchor targeted the old micro-benchmarks-benchmarks slug (verified via grep).
• Quickstart block — added a python benchmarks/bench_rf.py line and a container-only ./utils/run_container.sh python benchmarks/bench_gpu.py --mode train --find-max line, plus a one-line pointer to the GPU-benchmark section of benchmarks/README.md.
• Purpose table — appended two rows (bench_rf.py, bench_gpu.py); existing four rows preserved verbatim.
• Single-process note — replaced with a note that only the four CPU micro-benchmarks are single-process (manager.n_processes-parallelized in the pipeline); bench_rf uses sklearn n_jobs=-1; bench_gpu reports aggregate throughput across --num-gpus MirroredStrategy replicas plus per-GPU peak VRAM.

Baseline tables intentionally not duplicated — the "Baseline numbers" opener at ~line 214 already points readers to benchmarks/README.md, and #266 explicitly said not to copy the Blackwell / bla0 tables here.

Per-file rationale for what this PR does not touch (all confirmed in #266):

• CLAUDE.md — never mentions benchmarks.
• CONTRIBUTING.md — only a directory-tree pointer with correct targets; the "micro" descriptor drift there isn't worth churning the caption.
• README.md, SECURITY.md — no relevant surface.
• KNOWN_ISSUES.md — no new entry needed for the --find-max 4096 int32 launch-config cap; it's benchmark-local and already documented in bench_gpu.py + benchmarks/README.md.
• benchmarks/README.md and every .py under benchmarks/ — already landed by feat(benchmarks): GPU benchmark for Beta-VAE train step + encoder inference #253.
• src/aetherscan/cli.py — untouched by feat(benchmarks): GPU benchmark for Beta-VAE train step + encoder inference #253, so no README CLI Reference regeneration.

SKILL.md edits (apply manually: .claude/ is read-only in CI)

.claude/skills/aetherscan-repo-context/SKILL.md Project Structure tree has a benchmarks/ pointer (~lines 140-141) describing the contents as "Standalone micro-benchmarks" — mildly wrong after #253 (the suite now includes a non-micro GPU benchmark). Apply Edit 1 VERBATIM locally.

Edit 1: update the benchmarks/ line in the Project Structure tree.

Find (exactly, two lines):

benchmarks/              # Standalone micro-benchmarks (not collected by pytest);
                         # see benchmarks/README.md + docs/BENCHMARKING.md

Replace with (three lines):

benchmarks/              # Standalone benchmarks — CPU micro-benchmarks + a GPU
                         # benchmark (bench_gpu.py, container-only); not collected
                         # by pytest. See benchmarks/README.md + docs/BENCHMARKING.md

No other lines in SKILL.md need to change — it does not enumerate individual benchmark scripts.

Human-apply steps

@zachtheyek — you're CODEOWNER (* @zachtheyek) and the only person who can land the SKILL.md change (Claude Code cannot write under .claude/ in CI). After this PR merges:

```bash
gh pr checkout

open .claude/skills/aetherscan-repo-context/SKILL.md

apply Edit 1 above (two-line block becomes three-line block)

git add .claude/skills/aetherscan-repo-context/SKILL.md
git commit -S -m "docs(skill): note bench_gpu.py in benchmarks/ tree caption"
git push
```

Test plan

• pre-commit run --files docs/BENCHMARKING.md — passes (ruff/format skipped, no python touched; end-of-files, trailing-whitespace, private-key, gitleaks all pass).
• Reviewer: skim the five edited regions in docs/BENCHMARKING.md for wording drift vs benchmarks/README.md.
• Reviewer: confirm no in-file anchor targeted micro-benchmarks-benchmarks (grep clean at author time).
• Reviewer: apply the manual SKILL.md edit above.

Closes #266.
Also closes #264 and #265 (stale placeholders; superseded by #266).

[claude]

Claude finished @claude[bot]'s task in 4m 2s —— View job

---

Claude PR review

• Fetch base + read the changed file (docs/BENCHMARKING.md)
• Cross-check wording against benchmarks/README.md and benchmarks/bench_gpu.py / bench_rf.py
• Confirm none of the deletions break in-file or cross-file anchors
• Verify the five edits called out in [DOCUMENTATION]: List new bench_gpu.py + bench_rf.py in docs/BENCHMARKING.md (PR #253) #266 all landed and nothing else drifted
• Post review

Overall: LGTM — tight, well-scoped follow-up. Two inline nits (posted on lines 172-174 and 202) and one out-of-PR observation below. Nothing blocking.

What's good

• All five edits promised in the PR body landed and match benchmarks/README.md on the load-bearing claims I spot-checked against source:
  • bench_gpu: aggregate throughput across MirroredStrategy replicas + per-GPU peak VRAM (benchmarks/bench_gpu.py:24-32).
  • bench_rf: sklearn n_jobs=-1, no TF dependency, framework-free (benchmarks/bench_rf.py:15-31, 45).
  • CLI flags (--mode, --num-gpus, --batch-sizes, --accumulation-steps, --find-max) all real.
  • bench_gpu is genuinely container-only; bench_rf isn't (matches how the quickstart block is split).
• The #gpu-benchmark cross-doc anchor at line 192 resolves cleanly (benchmarks/README.md:87).
• Section rename Micro-benchmarks (benchmarks/) → Standalone benchmarks (benchmarks/) is safe: no inbound anchor targeted the old micro-benchmarks-benchmarks slug (grep -rn 'BENCHMARKING\.md#' returns zero anchor hits; only bare-file references).
• Purpose-table rows preserved verbatim; two new rows appended cleanly.
• Minor tidy on line 184 (dropped one stray space before the bench_pfb_vs_spline.py comment so the column aligns with the new bench_rf.py row).

Inline notes

• docs/BENCHMARKING.md:172-174 — mild "CPU micro-benchmarks" set-membership drift (5 in intro vs 4 in the closing note); posted a drop-in reshape that makes both usages consistent.
• docs/BENCHMARKING.md:202 — the parenthetical train.train_random_forest in the new bench_rf.py purpose row reads as a dotted stage name but isn't one (this file's own stage-instrumentation table at line 82 documents the umbrella span as train.rf). Suggested two disambiguations.

One out-of-PR observation (not for this PR)

docs/README.md:8 still describes the suite as "the benchmarks/ micro-benchmarks", which goes stale for the same reason docs/BENCHMARKING.md did — the suite now includes bench_gpu, which isn't a micro-benchmark. The PR body's "does not touch" list enumerates README.md, SECURITY.md, KNOWN_ISSUES.md, and CONTRIBUTING.md, but not docs/README.md, so worth deciding whether to fold into a small follow-up (or explicitly out-of-scope). pyproject.toml:179 and .gitignore:12 also say "micro-benchmark", but those are internal comments and fine to leave. CONTRIBUTING.md:211-212's tree caption is the same "micro-benchmarks" wording, which the PR body already flags as intentionally not worth churning — same call applies here, just noting for completeness.

On the SKILL.md manual edit

The .claude/skills/aetherscan-repo-context/SKILL.md Edit 1 in the PR body is well-specified (exact two-line find, three-line replace, no other edits). Correct call to defer — the .claude/ tree is CI-read-only, and the CODEOWNER-driven manual apply is the sanctioned path. No feedback on that edit itself.
· Branch: claude/docs-bench-followup-266

[claude]

Nit — mild set-membership drift for "CPU micro-benchmarks."

This paragraph groups bench_rf into "CPU micro-benchmarks" (5 scripts). The note near the bottom of the same section (lines 205-208) then says "Only the four CPU micro-benchmarks (bench_normality, bench_injection, bench_lognorm_downsample, bench_pfb_vs_spline) are single-process" — implicitly excluding bench_rf. Both readings coexist in the file.

A reader following the text sequentially won't be tripped up, but anyone grepping for "CPU micro-benchmarks" will see two subsets with the same label. One tightening (drop-in replacement for the two-sentence "Most are CPU… bench_gpu is a real-GPU profiler…" span) that makes the sets consistent with the closing note:

Suggested change

	one can be measured in seconds instead of via a full run. Most are CPU micro-benchmarks
	(`bench_normality`, `bench_injection`, `bench_lognorm_downsample`, `bench_pfb_vs_spline`,
	`bench_rf`) that print ops/s; `bench_gpu` is a real-GPU profiler for the Beta-VAE that reports
	Small standalone scripts that time individual pipeline kernels in isolation, so a change to
	one can be measured in seconds instead of via a full run. Most are CPU: four micro-benchmarks
	(`bench_normality`, `bench_injection`, `bench_lognorm_downsample`, `bench_pfb_vs_spline`) that
	print ops/s, plus `bench_rf` (Random Forest, sklearn `n_jobs=-1`); `bench_gpu` is a real-GPU
	profiler for the Beta-VAE that reports throughput + peak VRAM across a per-replica batch-size
	sweep (NGC-container-only). **Not**

The TL;DR bullet at lines 28-32 has the same "Most are CPU (…including bench_rf)" phrasing and would benefit from the same reshape for parallelism, but it's less load-bearing since it's a summary.

[claude]

Nit — train.train_random_forest reads as a dotted stage name but isn't one.

The "What is instrumented" table above (line 82) documents the actual umbrella span as train.rf (with .data_generation, .encode, .fit sub-stages). The parenthetical here looks like the same dotted-stage vocabulary but is actually the Python function name (train.py::train_random_forest), which reads as inconsistent within the same file. Two ways to disambiguate:

• If you meant the stage: (`train.rf` / `train.rf.fit`).
• If you meant the function: (`train_random_forest` in `train.py`) or (`train.py::train_random_forest`).

benchmarks/README.md:40 has the same wording, so worth aligning both in a follow-up (out of this PR's scope) — but the collision with the stage table is only jarring in docs/BENCHMARKING.md where the stage table sits ~120 lines above the row.

[claude]

Internal change — no release note.