diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 56ad73f..ad95c03 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -29,6 +29,11 @@ jobs: # VIRTUAL_ENV, so install straight into it — no `uv venv` needed. run: uv pip install -e ".[dev]" + - name: Formatting and lint (ruff) + run: | + uvx ruff format --check . + uvx ruff check . + - name: Run the test suite run: .venv/bin/pytest diff --git a/CLAUDE.md b/CLAUDE.md index 9cf3907..7ee3a37 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -8,7 +8,13 @@ itself lives in `skills/skill-validator/`; its deterministic engine is in ## Development +- **Setup:** `uv venv && uv pip install -e ".[dev]"` — installs the `skval` + console entrypoint. - **Tests:** `uv run pytest` — deterministic, no network or model calls. + Single file/test: `uv run pytest tests/test_scoring.py -k `. +- **CLI:** `uv run skval `. + `structural` is the no-model-calls path; `estimate` previews token/$ cost of a + full run without spending anything. - **Self-validation gate (run before every PR):** ```bash uv run python skills/skill-validator/scripts/validate_structural.py skills/skill-validator --out /tmp/skval-self @@ -19,6 +25,37 @@ itself lives in `skills/skill-validator/`; its deterministic engine is in (`.claude/hooks/format.sh`) reformats `.py` files on save, so keep edits ruff-clean. +## Architecture + +Two tiers, split by whether a model is needed: + +- **Deterministic engine** (`skills/skill-validator/scripts/`): plain Python, + unit-tested, no network/model calls. Scores D1 (structural) + D6 (safety gate) + and does all math/assembly. Flow: `resolve_skill` (dir / SKILL.md / .skill → + canonical dir, hardened against zip-slip/symlinks/bombs) → `static_checks` + + `safety_scan` → `scoring` (weighted composite; **safety is a veto gate, not a + weight**) → `scorecard` (renders json + md). +- **Model-driven stages** (D2–D5): not code — prompt guides in + `skills/skill-validator/agents/*.md` (executor, grader, artifact-judge, + triggering, user-simulator, comparator, eval-generator), orchestrated by + `SKILL.md` via subagents. They drop artifacts into a workspace + (`grading.json`, `artifact_judgment.json`, `triggering.json`) which + `validate_full.py` aggregates (`stats.py` pass^k / error bars, + `aggregate.py`, `dimensions.py` signal→dimension mapping) into the same + scorecard shape. + +Conventions: + +- `scripts/` modules are **flat top-level imports** (`import scoring`, no + package prefix) — `[tool.setuptools.package-dir]` in `pyproject.toml` maps + the dir to the root, and pytest's `pythonpath` mirrors it. A new module must + be added to `py-modules` there. +- `skills/skill-validator/references/scoring-rubric.md` is the **source of + truth** for weights and grade bands; code and docs must match it. +- `classify.py` types a skill (task / file_transform / interactive / + discipline / reference) and that type routes the eval strategy — see the + table in `SKILL.md`. + ## UI / Design (the landing page in `docs/`) - The site is a single, hand-tuned static page: `docs/index.html` (dark theme — diff --git a/README.md b/README.md index 397041d..6950eaf 100644 --- a/README.md +++ b/README.md @@ -116,7 +116,7 @@ tests/ # pytest suite + fixtures (good / bad / unsafe skills) ## Develop ```bash -uv run pytest # 163 tests, deterministic, no network/model calls +uv run pytest # deterministic, no network/model calls ``` Built with the [Superpowers](https://github.com/obra/superpowers) methodology