Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
.orchestration/
.dev-loop/
node_modules/
.DS_Store
*.log
Expand Down
6 changes: 6 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,7 @@ confidence: verified | field-tested | unverified
sources:
- <url or citation>
last_verified: YYYY-MM-DD
verified_model: <model-id> # optional
related: [<page id>, ...]
---
```
Expand All @@ -133,6 +134,11 @@ related: [<page id>, ...]
measurement. `field-tested`: worked in real production use; context described in the
page. `unverified`: candidate knowledge; lint reports it until upgraded or removed.

`verified_model` (optional) names the model generation the page's guidance was
verified against (e.g. `claude-fable-5`); model-coupled pages missing it, or
carrying an outdated one, are surfaced by lint as re-verification candidates —
see `scripts/wiki-lint-model-era.js`.

### Section skeleton

```markdown
Expand Down
43 changes: 43 additions & 0 deletions plans/wiki-model-era/analysis.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
# Analysis — wiki-model-era

<!--
Parsed by skills/wiki-plan/scripts/plan-gate.sh (gate-A). Keep section
headers exactly as shown.
-->

## Requirements
| Rule | Concrete example | Open question |
|------|------------------|---------------|
| R1: An optional frontmatter field records the model context a page's guidance was verified against; documented in AGENTS.md + templates/page.md; NOT added to structure-check REQUIRED_KEYS (absence is a signal, not an error) | Given a page with `verified_model: claude-opus-4-5` in frontmatter, when wiki-structure-checks runs, then no new finding; given a page without the field, then structure-checks still exits 0 | |
| R2: A mechanical script reports re-verification candidates: pages whose body is model-coupled (keyword match) AND (field absent OR field value matches no current-generation token) | Given a fixture page whose body says "Claude returns…" and no `verified_model`, when `node scripts/wiki-lint-model-era.js <root>` runs, then stderr lists `revalidate:<file>: model-coupled, no verified_model` and exit is 3 | |
| R3: The current-generation token set is overridable (CLI flag wins over env var over built-in default) so the default's aging never requires editing callers | Given `--current opus-4,fable-5` and a page with `verified_model: claude-sonnet-3-7`, then that page is listed; given `verified_model: claude-fable-5`, then it is not | |
| R4: wiki-lint SKILL.md gains the new check row (info severity) and the health score stays consistent (total_weight 24 → 25); the bats pin in tests/wiki-lint-score.bats is bumped in the same task | Given the updated SKILL.md, when `bats tests/wiki-lint-score.bats` runs, then all tests pass with the new `total_weight = 25` assertion | |
| R5: The new script has bats tests covering normal + error + boundary cases with fixtures (per tests/fixtures/prohibitions convention) | Given `bats tests/wiki-lint-model-era.bats`, then ≥3 cases pass incl. ≥1 error case (bad usage → exit 4) and ≥1 boundary (empty dir → exit 0) | |
| R6: No retroactive edit of the live corpus, and CI stays green: the new script is NOT added to test.yml's blocking wiki-checks step (live corpus has ~27 candidates by design) | Given the final diff, when `node scripts/wiki-structure-checks.js wiki && node scripts/wiki-lint-prohibitions.js wiki` and the full bats suite run, then rc=0 and no wiki/ page content changed | |

## Ground truth
- Baseline: bats tests/wiki-lint-score.bats tests/wiki-structure-checks.bats tests/wiki-lint-prohibitions.bats -> rc=0, HEAD 300e4bef52fb9338b7d87df4c59851e09bd6512a, git status clean

### Affected files
- scripts/wiki-lint-model-era.js (new) — evidence: `ls scripts/ | grep model-era` -> 0 hits (name free)
- tests/wiki-lint-model-era.bats (new) + tests/fixtures/model-era/ (new) — evidence: `ls tests/ tests/fixtures/` -> no model-era entries; prohibitions fixture convention confirmed (`ls tests/fixtures/prohibitions` -> bad.md, good.md)
- templates/page.md — evidence: `grep -n last_verified templates/page.md` -> 1 hit (frontmatter block to extend)
- AGENTS.md — evidence: `grep -n "### Frontmatter" AGENTS.md` -> line 116 (frontmatter schema section to extend)
- skills/wiki-lint/SKILL.md — evidence: `grep -n "total_weight = 24" skills/wiki-lint/SKILL.md` -> 1 hit (Checks table + Health score section)
- tests/wiki-lint-score.bats — evidence: `grep -n "total_weight = 24" tests/wiki-lint-score.bats` -> 1 hit (prose pin to bump)

## Constraints
- tests/wiki-lint-score.bats pins SKILL.md health-score constants (`total_weight = 24`, weight rows, `health: NN/100` format) — checked: `grep -n "total_weight" tests/wiki-lint-score.bats` -> pinned; bumping the weight requires bumping this pin in the same task (T3)
- .github/workflows/test.yml runs `node scripts/wiki-structure-checks.js wiki` and `node scripts/wiki-lint-prohibitions.js wiki` as blocking — checked: `grep -n wiki-structure-checks .github/workflows/test.yml` -> lines 44-47; the new script must NOT join this blocking step (live corpus would exit 3 by design)
- scripts/wiki-structure-checks.js REQUIRED_KEYS (line 49) must NOT gain the new field — checked: `grep -n REQUIRED_KEYS scripts/wiki-structure-checks.js`; an unknown extra frontmatter key produces no finding in that script (verified by reading its checks — only missing/duplicate/stray keys are flagged), so adding `verified_model` to fixtures/pages is compatible
- AGENTS.md is schema-layer ("change only with repo owner approval") — the repo owner requested this feature in this session

## Spikes
- Quirk-phrase detection is the wrong mechanical signal: explicit "model tends/may/will" phrasing appears on only 3 pages, while model-coupled *subject* keywords identify the real aging surface — `claude` 23 pages, `llm` 16, `subagent` 5, `opus` 2, `context window` 2, `sonnet` 1, `gpt-` 1, `hallucinat` 0; union ≈ 27 of 271 pages (10%), concentrated in wiki/platforms/tools and wiki/backend/common/llm. Verified by grep over the live corpus this session. → Detection = case-insensitive word-boundary keyword match on the body (excluding `## Sources`, same scope convention as wiki-lint-prohibitions.js).
- Exit-code conventions differ between existing scripts (prohibitions: 2 for bad dir; structure-checks: 0/3/4 documented). → Follow the newer documented 0 clean / 3 findings / 4 usage contract of wiki-structure-checks.js.
- ~27 candidates on the live corpus is an actionable list (not noise) for an info-severity report-only check; the health score counts check pass/fail, not finding count, so the score impact is bounded to the one new info check.

## Research
| Query | Source | Applied |
|-------|--------|---------|
| pruning outdated model-specific instructions CLAUDE.md steering files agent context rot model upgrade | brave-search MCP: jonkrohn.com field guide ("Treat steering files as code: owned, reviewed and pruned"), r/ClaudeCode on Opus 5 ("anything correcting model or harness specific behaviour doesn't belong… Opus 5 does not behave the same as Sonnet 5"), addyosmani.com self-improving agents (archive obsolete info; pruned knowledge files), windowsforum/Anthropic guidance (prune stale CLAUDE.md) | Confirms the feature premise: model/harness-corrective guidance ages with model generations and needs a review trigger, not silent accumulation. Grounds D2 (field records the verification-time model) and D5 (report-only info severity — pruning is a human review act, not an auto-fix) |
35 changes: 35 additions & 0 deletions plans/wiki-model-era/design.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# Design — wiki-model-era

<!--
Parsed by skills/wiki-plan/scripts/plan-gate.sh (gate-B). Keep section
headers exactly as shown.
-->

## Decisions
| # | Decision | Choice | Wiki basis | Rejected alternative | Testability |
|---|----------|--------|------------|----------------------|-------------|
| D1 | Mechanical/semantic split for the new check | Countable half (keyword match, field presence/value) in a standalone `scripts/wiki-lint-model-era.js`; semantic half (whether a flagged page truly needs re-verification, what to rewrite) stays in wiki-lint SKILL.md's fix protocol — same split as prohibitions/structure-checks | wiki/qa/process/llm-review-pipelines.md | Prompt-prose-only check inside SKILL.md — undecidable drift, no CI-comparable output | tests/wiki-lint-model-era.bats runs the script on fixtures; SKILL.md check row names the script |
| D2 | Frontmatter field | New OPTIONAL key `verified_model: <model-id>` (single freeform id, e.g. `claude-opus-4-5`), placed after `last_verified`; NOT added to structure-checks REQUIRED_KEYS; absence on a model-coupled page = re-verification candidate signal, never an error | [no-wiki] | Overloading `last_verified` (date only) — date staleness is already check 8 and misses model-generation coupling; a `models:` list — speculative flexibility nothing consumes | R1 example: structure-checks exits 0 on pages with and without the field |
| D3 | Model-coupled page detection | Case-insensitive, word-boundary keyword set — `claude`, `sonnet`, `opus`, `gpt-`, `llm`, `subagent`, `hallucinat`, `context window` — matched over the page body EXCLUDING the `## Sources` section (scope-by-location, same exclusion wiki-lint-prohibitions.js uses); measured union ≈27/271 live pages | wiki/testing/quality/guard-shape-vs-consequence.md | Quirk-phrase regex ("model tends/may/will …") — measured 3/271 hits, misses the real aging surface (platforms/tools, backend/common/llm) | Fixture pair per Semgrep ruleid/ok split: a coupled page must be flagged, an uncoupled page and a Sources-only mention must not |
| D4 | Current-generation set + override precedence | CLI `--current <csv>` > env `DEV_LOOP_CURRENT_MODELS` (csv) > built-in default `opus-4,fable-5`; a page is current when ANY token is a case-insensitive substring of its `verified_model` value; the default lives in one documented constant at the top of the script | [no-wiki] | Hardcoded list with no override — recreates the exact aging the check exists to catch; config file — a third config surface for one value | Fixture stamped `verified_model: claude-sonnet-3-7` flips out of candidates under `--current sonnet-3`, back in under the default |
| D5 | Script output/exit contract + CI wiring | Exit 0 = no candidates, 3 = candidates, 4 = usage/unreadable root (structure-checks contract; distinct exit per outcome); stdout one summary line `pages: N, model-coupled: M, candidates: K`; stderr one line per candidate `revalidate:<file>: <reason>` where reason ∈ {`model-coupled, no verified_model`, `verified_model '<v>' not in current set`}; NOT added to test.yml's blocking wiki-checks step (live corpus carries ~27 candidates by design) | wiki/testing/quality/checks-that-cannot-pass.md | Wiring into CI blocking — permanent red on a report-only signal; boolean exit — conflates "could not run" with "found candidates" | bats: empty dir → 0, bad usage → 4, candidate fixture → 3; `git diff .github/` empty |
| D6 | wiki-lint SKILL.md integration | New check row `#12` (info severity): "model-coupled page whose `verified_model` is absent or outside the current generation — reported by `node scripts/wiki-lint-model-era.js`; re-verification candidate list, report-only"; Phase 0 gains the script as a fourth baseline command; Health score: info checks become 10–12, `total_weight = 25` (4×3 + 5×2 + 3×1); the prose pins in tests/wiki-lint-score.bats (`total_weight = 24` and Phase 0 three-command family assertion) are bumped in the SAME task | wiki/qa/document-verification/editing-a-gated-document.md | Leaving the score at 24 — Checks table and score section disagree; a new `warn` severity — a report-only candidate list is not a defect claim | bats tests/wiki-lint-score.bats green after the edit; anchor inventory greps recorded in the task |
| D7 | Test design for the new script | Fixtures under `tests/fixtures/model-era/` (convention: tests/fixtures/prohibitions); every @test puts its deciding assertion as the final command (bash 3.2 masking, issue #114 header convention); ≥1 negative control per detection direction; case set = normal (coupled+unstamped flagged) + error (usage → 4) + boundary (empty dir → 0; Sources-only mention → not flagged; stamped-current → not flagged; stamped-old → flagged; --current/env override) | wiki/testing/quality/tests-that-cannot-fail.md | Inline heredoc fixtures — diverges from the established fixture convention; corpus-dependent tests over live wiki/ — breaks on every ingest | bats tests/wiki-lint-model-era.bats — each check has a fixture that must redden it (seeded fault) |
| D8 | Schema documentation surface | AGENTS.md `### Frontmatter` block gains the `verified_model` line + one meaning sentence ("optional: the model generation the page's guidance was verified against; model-coupled pages missing it are surfaced by lint as re-verification candidates"); templates/page.md frontmatter gains the same line with a placeholder comment; wiki-lint SKILL.md is NOT the schema authority (AGENTS.md is the schema layer) | [no-wiki] | Documenting only in SKILL.md — schema truth would live in a workflow file; a new REQUIRED key — retroactively invalidates 271 pages against R6 | grep for `verified_model` in AGENTS.md + templates/page.md; structure-checks still 0 findings on live corpus |

Note on D6 arithmetic: weights are error 3×4 checks + warn 2×5 checks + info 1×3 checks = 12+10+3 = **25**; `total_weight = 25`.

`[no-wiki]` decisions (D2 wiki-schema field design, D4 override precedence, D8 schema-layer authority) are ingest candidates — genuinely uncovered conventions per the reviewer.

## Review

plan-reviewer subagent, call 1 of 1 (2026-09-03):

VERDICT: PASS
FINDINGS (all non-blocking):
- R1–R6 each map to at least one Decision row (R1→D2/D8, R2→D1/D3/D5, R3→D4, R4→D6, R5→D7, R6→D5); no uncovered rule.
- All five non-[no-wiki] Wiki basis paths exist under the wiki root and substantively support their decisions.
- [no-wiki] tags verified as genuinely uncovered conventions, not grounding dodges.
- Verified against the live repo: REQUIRED_KEYS has no unknown-key check (D2/R1 holds); exit contract 0/3/4 matches structure-checks; total_weight 24→25 arithmetic correct; test.yml blocking step confirmed limited to the two existing scripts.
- No Decision contradicts analysis.md Constraints; every Rejected alternative concretely justified (D3 backed by measured 3/271 vs 27/271 grep).
SUMMARY: All requirements covered, groundings resolve, constraints hold, baseline reproduced green. PASS.
33 changes: 33 additions & 0 deletions plans/wiki-model-era/plan.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# wiki-model-era

Goal: The wiki gains an aging-management signal for model-generation debt: an
optional `verified_model` frontmatter field records the model a page's guidance
was verified against, and a mechanical checker reports model-coupled pages whose
field is absent or outdated as re-verification candidates in the wiki-lint
report. Acceptance: all six R-rules in analysis.md hold; live corpus unedited;
CI stays green.

Stack: Node ≥18 (repo scripts convention, no deps), bats (tests convention),
Markdown/YAML frontmatter (wiki schema).

## Decisions
| # | Decision | Choice | Wiki basis |
|---|----------|--------|------------|
| D1 | Mechanical/semantic split | Countable half in `scripts/wiki-lint-model-era.js`; semantic half in wiki-lint SKILL.md fix protocol | wiki/qa/process/llm-review-pipelines.md |
| D2 | Frontmatter field | Optional `verified_model: <model-id>` after `last_verified`; never a REQUIRED key | [no-wiki] |
| D3 | Model-coupled detection | Case-insensitive keyword set (claude, sonnet, opus, gpt-, llm, subagent, hallucinat, context window), leading boundary `(?<![A-Za-z0-9-])`, body excluding frontmatter + `## Sources` + markdown link URLs; index.md files skipped | wiki/testing/quality/guard-shape-vs-consequence.md |
| D4 | Current-generation set | CLI `--current <csv>` > env `DEV_LOOP_CURRENT_MODELS` > default `opus-4,fable-5`; current = any token is a case-insensitive substring of the field value | [no-wiki] |
| D5 | Output/exit contract + CI | Exit 0 clean / 3 candidates / 4 usage; stdout `pages: N, model-coupled: M, candidates: K`; stderr `revalidate:<file>: <reason>`; NOT in test.yml blocking step | wiki/testing/quality/checks-that-cannot-pass.md |
| D6 | SKILL.md integration | Check #12 (info), Phase 0 fourth command, `total_weight = 25`, score pins in tests/wiki-lint-score.bats bumped same task | wiki/qa/document-verification/editing-a-gated-document.md |
| D7 | Test design | Fixtures under tests/fixtures/model-era/ (mini-root per scenario); deciding assertion last per @test; ≥1 negative control per direction | wiki/testing/quality/tests-that-cannot-fail.md |
| D8 | Schema docs | AGENTS.md frontmatter block + meaning sentence; templates/page.md YAML comment line; SKILL.md is not schema authority | [no-wiki] |

## Size verdict
size: small

## Task order
| Task | Depends on | Parallel-ok |
|------|-----------|-------------|
| 01-model-era-script | — | parallel-ok with 02 |
| 02-schema-docs | — | parallel-ok with 01 |
| 03-skill-check-row | 01 | — |
1 change: 1 addition & 0 deletions plans/wiki-model-era/review-verdict.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
VERDICT: PASS
Loading
Loading