From 59e083d0800024669ce34daa1b7dae070c6cceb4 Mon Sep 17 00:00:00 2001 From: Claude Date: Thu, 2 Jul 2026 13:21:34 -0300 Subject: [PATCH] docs(status): 2026-07-02 session recap --- docs/PROJECT_STATUS.md | 47 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100644 docs/PROJECT_STATUS.md diff --git a/docs/PROJECT_STATUS.md b/docs/PROJECT_STATUS.md new file mode 100644 index 0000000..f4348b5 --- /dev/null +++ b/docs/PROJECT_STATUS.md @@ -0,0 +1,47 @@ +# Project status + +A dated logbook of where skval stands at the end of each work session. Newest entry +first. For usage see [USAGE.md](USAGE.md); for architecture see the repo `CLAUDE.md`. + +## 2026-07-02 - Project review, competitor research, and D2 statistics upgrade + +**Where we were:** skval's deterministic engine, six-dimension scoring, and landing +page were all shipped and green (163 tests, self-validation 100/A/Ship). The +`CLAUDE.md` documented commands and design but not setup, the CLI surface, or the +two-tier architecture, and CI ran tests without enforcing formatting/lint. + +**What we did:** +- Expanded `CLAUDE.md` (setup/install, `skval` CLI subcommands, single-test command, + and an architecture section: deterministic engine vs model-driven agent stages, + flat `scripts/` imports, rubric as source of truth); added a `ruff format --check` + + `ruff check` step to CI; dropped the hardcoded test count in the README (#48). +- Ran a deep-research survey of similar eval frameworks (105 agents, 23 sources, 25 + adversarially verified claims). Key result: skval's core design is externally + validated (SkillsBench for lift-vs-baseline, tau-bench for pass^k), and the highest + value-per-effort gaps are statistical. Full findings are in the 2026-07-02 chat. +- Implemented the top two research items on the D2 effectiveness dimension (#49): + Hake **normalized gain** `g = (skill − baseline) / (1 − baseline)` reported in the + scorecard, and **paired-difference significance** (Miller 2024) that pairs per-eval + means when ≥2 evals exist in both configs, falling back to the unpaired SE for one + eval. New pure helpers `stats.normalized_gain` / `stats.paired_diff`, TDD, +6 tests + (now 169 passing, ruff clean, self-validation still 100/A/Ship). + +**Decisions:** kept `baseline_lift` and its unpaired SE unchanged (new fields are +additive) so nothing downstream breaks. Deferred the Bayesian/Bayes@N verdict rework +(research item 3) to its own design pass because it changes what the score *means* - +not a drop-in like normalized gain was. + +**Pending / next:** (from the research backlog, roughly by value-per-effort) +- [ ] Item 4 - named deterministic safety checks in `safety_scan.py` (tool over-grant, + unpinned deps, `eval`/`exec`, `curl | sh`), per OWASP Agentic / SkillCheck. Best + next PR: cheap greps, deterministic, high signal. +- [ ] Item 5 - score cost/latency, not just record them: `runs_io` already loads + token/time metrics; surface actual spend per config in the scorecard. +- [ ] Item 3 - Bayes@N posterior verdicts with credible intervals + adaptive trial + allocation. Needs a design pass first (pass^k all-k vs pass@k semantics, how + credible intervals map onto the 0-100 bands and Ship/Revise/Reject). +- [ ] Item 6 - Agent-as-a-Judge (multi-agent) for D4, or route more of D4 through + deterministic end-state checks where ground truth is checkable. +- [ ] Item 7 - description-optimization loop when D5 triggering scores low. +- [ ] Follow-up research: the mainstream-framework half (promptfoo/DeepEval/Braintrust + CI gating + reporting UX) produced no verified claims - worth a targeted pass.