diff --git a/.gitignore b/.gitignore index cbb4669..67b8a88 100644 --- a/.gitignore +++ b/.gitignore @@ -1,4 +1,5 @@ .orchestration/ +.dev-loop/ node_modules/ .DS_Store *.log diff --git a/AGENTS.md b/AGENTS.md index 2fb8234..25a87e7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -125,6 +125,7 @@ confidence: verified | field-tested | unverified sources: - last_verified: YYYY-MM-DD +verified_model: # optional related: [, ...] --- ``` @@ -133,6 +134,11 @@ related: [, ...] measurement. `field-tested`: worked in real production use; context described in the page. `unverified`: candidate knowledge; lint reports it until upgraded or removed. +`verified_model` (optional) names the model generation the page's guidance was +verified against (e.g. `claude-fable-5`); model-coupled pages missing it, or +carrying an outdated one, are surfaced by lint as re-verification candidates — +see `scripts/wiki-lint-model-era.js`. + ### Section skeleton ```markdown diff --git a/plans/wiki-model-era/analysis.md b/plans/wiki-model-era/analysis.md new file mode 100644 index 0000000..857979d --- /dev/null +++ b/plans/wiki-model-era/analysis.md @@ -0,0 +1,43 @@ +# Analysis — wiki-model-era + + + +## Requirements +| Rule | Concrete example | Open question | +|------|------------------|---------------| +| R1: An optional frontmatter field records the model context a page's guidance was verified against; documented in AGENTS.md + templates/page.md; NOT added to structure-check REQUIRED_KEYS (absence is a signal, not an error) | Given a page with `verified_model: claude-opus-4-5` in frontmatter, when wiki-structure-checks runs, then no new finding; given a page without the field, then structure-checks still exits 0 | | +| R2: A mechanical script reports re-verification candidates: pages whose body is model-coupled (keyword match) AND (field absent OR field value matches no current-generation token) | Given a fixture page whose body says "Claude returns…" and no `verified_model`, when `node scripts/wiki-lint-model-era.js ` runs, then stderr lists `revalidate:: model-coupled, no verified_model` and exit is 3 | | +| R3: The current-generation token set is overridable (CLI flag wins over env var over built-in default) so the default's aging never requires editing callers | Given `--current opus-4,fable-5` and a page with `verified_model: claude-sonnet-3-7`, then that page is listed; given `verified_model: claude-fable-5`, then it is not | | +| R4: wiki-lint SKILL.md gains the new check row (info severity) and the health score stays consistent (total_weight 24 → 25); the bats pin in tests/wiki-lint-score.bats is bumped in the same task | Given the updated SKILL.md, when `bats tests/wiki-lint-score.bats` runs, then all tests pass with the new `total_weight = 25` assertion | | +| R5: The new script has bats tests covering normal + error + boundary cases with fixtures (per tests/fixtures/prohibitions convention) | Given `bats tests/wiki-lint-model-era.bats`, then ≥3 cases pass incl. ≥1 error case (bad usage → exit 4) and ≥1 boundary (empty dir → exit 0) | | +| R6: No retroactive edit of the live corpus, and CI stays green: the new script is NOT added to test.yml's blocking wiki-checks step (live corpus has ~27 candidates by design) | Given the final diff, when `node scripts/wiki-structure-checks.js wiki && node scripts/wiki-lint-prohibitions.js wiki` and the full bats suite run, then rc=0 and no wiki/ page content changed | | + +## Ground truth +- Baseline: bats tests/wiki-lint-score.bats tests/wiki-structure-checks.bats tests/wiki-lint-prohibitions.bats -> rc=0, HEAD 300e4bef52fb9338b7d87df4c59851e09bd6512a, git status clean + +### Affected files +- scripts/wiki-lint-model-era.js (new) — evidence: `ls scripts/ | grep model-era` -> 0 hits (name free) +- tests/wiki-lint-model-era.bats (new) + tests/fixtures/model-era/ (new) — evidence: `ls tests/ tests/fixtures/` -> no model-era entries; prohibitions fixture convention confirmed (`ls tests/fixtures/prohibitions` -> bad.md, good.md) +- templates/page.md — evidence: `grep -n last_verified templates/page.md` -> 1 hit (frontmatter block to extend) +- AGENTS.md — evidence: `grep -n "### Frontmatter" AGENTS.md` -> line 116 (frontmatter schema section to extend) +- skills/wiki-lint/SKILL.md — evidence: `grep -n "total_weight = 24" skills/wiki-lint/SKILL.md` -> 1 hit (Checks table + Health score section) +- tests/wiki-lint-score.bats — evidence: `grep -n "total_weight = 24" tests/wiki-lint-score.bats` -> 1 hit (prose pin to bump) + +## Constraints +- tests/wiki-lint-score.bats pins SKILL.md health-score constants (`total_weight = 24`, weight rows, `health: NN/100` format) — checked: `grep -n "total_weight" tests/wiki-lint-score.bats` -> pinned; bumping the weight requires bumping this pin in the same task (T3) +- .github/workflows/test.yml runs `node scripts/wiki-structure-checks.js wiki` and `node scripts/wiki-lint-prohibitions.js wiki` as blocking — checked: `grep -n wiki-structure-checks .github/workflows/test.yml` -> lines 44-47; the new script must NOT join this blocking step (live corpus would exit 3 by design) +- scripts/wiki-structure-checks.js REQUIRED_KEYS (line 49) must NOT gain the new field — checked: `grep -n REQUIRED_KEYS scripts/wiki-structure-checks.js`; an unknown extra frontmatter key produces no finding in that script (verified by reading its checks — only missing/duplicate/stray keys are flagged), so adding `verified_model` to fixtures/pages is compatible +- AGENTS.md is schema-layer ("change only with repo owner approval") — the repo owner requested this feature in this session + +## Spikes +- Quirk-phrase detection is the wrong mechanical signal: explicit "model tends/may/will" phrasing appears on only 3 pages, while model-coupled *subject* keywords identify the real aging surface — `claude` 23 pages, `llm` 16, `subagent` 5, `opus` 2, `context window` 2, `sonnet` 1, `gpt-` 1, `hallucinat` 0; union ≈ 27 of 271 pages (10%), concentrated in wiki/platforms/tools and wiki/backend/common/llm. Verified by grep over the live corpus this session. → Detection = case-insensitive word-boundary keyword match on the body (excluding `## Sources`, same scope convention as wiki-lint-prohibitions.js). +- Exit-code conventions differ between existing scripts (prohibitions: 2 for bad dir; structure-checks: 0/3/4 documented). → Follow the newer documented 0 clean / 3 findings / 4 usage contract of wiki-structure-checks.js. +- ~27 candidates on the live corpus is an actionable list (not noise) for an info-severity report-only check; the health score counts check pass/fail, not finding count, so the score impact is bounded to the one new info check. + +## Research +| Query | Source | Applied | +|-------|--------|---------| +| pruning outdated model-specific instructions CLAUDE.md steering files agent context rot model upgrade | brave-search MCP: jonkrohn.com field guide ("Treat steering files as code: owned, reviewed and pruned"), r/ClaudeCode on Opus 5 ("anything correcting model or harness specific behaviour doesn't belong… Opus 5 does not behave the same as Sonnet 5"), addyosmani.com self-improving agents (archive obsolete info; pruned knowledge files), windowsforum/Anthropic guidance (prune stale CLAUDE.md) | Confirms the feature premise: model/harness-corrective guidance ages with model generations and needs a review trigger, not silent accumulation. Grounds D2 (field records the verification-time model) and D5 (report-only info severity — pruning is a human review act, not an auto-fix) | diff --git a/plans/wiki-model-era/design.md b/plans/wiki-model-era/design.md new file mode 100644 index 0000000..2556cad --- /dev/null +++ b/plans/wiki-model-era/design.md @@ -0,0 +1,35 @@ +# Design — wiki-model-era + + + +## Decisions +| # | Decision | Choice | Wiki basis | Rejected alternative | Testability | +|---|----------|--------|------------|----------------------|-------------| +| D1 | Mechanical/semantic split for the new check | Countable half (keyword match, field presence/value) in a standalone `scripts/wiki-lint-model-era.js`; semantic half (whether a flagged page truly needs re-verification, what to rewrite) stays in wiki-lint SKILL.md's fix protocol — same split as prohibitions/structure-checks | wiki/qa/process/llm-review-pipelines.md | Prompt-prose-only check inside SKILL.md — undecidable drift, no CI-comparable output | tests/wiki-lint-model-era.bats runs the script on fixtures; SKILL.md check row names the script | +| D2 | Frontmatter field | New OPTIONAL key `verified_model: ` (single freeform id, e.g. `claude-opus-4-5`), placed after `last_verified`; NOT added to structure-checks REQUIRED_KEYS; absence on a model-coupled page = re-verification candidate signal, never an error | [no-wiki] | Overloading `last_verified` (date only) — date staleness is already check 8 and misses model-generation coupling; a `models:` list — speculative flexibility nothing consumes | R1 example: structure-checks exits 0 on pages with and without the field | +| D3 | Model-coupled page detection | Case-insensitive, word-boundary keyword set — `claude`, `sonnet`, `opus`, `gpt-`, `llm`, `subagent`, `hallucinat`, `context window` — matched over the page body EXCLUDING the `## Sources` section (scope-by-location, same exclusion wiki-lint-prohibitions.js uses); measured union ≈27/271 live pages | wiki/testing/quality/guard-shape-vs-consequence.md | Quirk-phrase regex ("model tends/may/will …") — measured 3/271 hits, misses the real aging surface (platforms/tools, backend/common/llm) | Fixture pair per Semgrep ruleid/ok split: a coupled page must be flagged, an uncoupled page and a Sources-only mention must not | +| D4 | Current-generation set + override precedence | CLI `--current ` > env `DEV_LOOP_CURRENT_MODELS` (csv) > built-in default `opus-4,fable-5`; a page is current when ANY token is a case-insensitive substring of its `verified_model` value; the default lives in one documented constant at the top of the script | [no-wiki] | Hardcoded list with no override — recreates the exact aging the check exists to catch; config file — a third config surface for one value | Fixture stamped `verified_model: claude-sonnet-3-7` flips out of candidates under `--current sonnet-3`, back in under the default | +| D5 | Script output/exit contract + CI wiring | Exit 0 = no candidates, 3 = candidates, 4 = usage/unreadable root (structure-checks contract; distinct exit per outcome); stdout one summary line `pages: N, model-coupled: M, candidates: K`; stderr one line per candidate `revalidate:: ` where reason ∈ {`model-coupled, no verified_model`, `verified_model '' not in current set`}; NOT added to test.yml's blocking wiki-checks step (live corpus carries ~27 candidates by design) | wiki/testing/quality/checks-that-cannot-pass.md | Wiring into CI blocking — permanent red on a report-only signal; boolean exit — conflates "could not run" with "found candidates" | bats: empty dir → 0, bad usage → 4, candidate fixture → 3; `git diff .github/` empty | +| D6 | wiki-lint SKILL.md integration | New check row `#12` (info severity): "model-coupled page whose `verified_model` is absent or outside the current generation — reported by `node scripts/wiki-lint-model-era.js`; re-verification candidate list, report-only"; Phase 0 gains the script as a fourth baseline command; Health score: info checks become 10–12, `total_weight = 25` (4×3 + 5×2 + 3×1); the prose pins in tests/wiki-lint-score.bats (`total_weight = 24` and Phase 0 three-command family assertion) are bumped in the SAME task | wiki/qa/document-verification/editing-a-gated-document.md | Leaving the score at 24 — Checks table and score section disagree; a new `warn` severity — a report-only candidate list is not a defect claim | bats tests/wiki-lint-score.bats green after the edit; anchor inventory greps recorded in the task | +| D7 | Test design for the new script | Fixtures under `tests/fixtures/model-era/` (convention: tests/fixtures/prohibitions); every @test puts its deciding assertion as the final command (bash 3.2 masking, issue #114 header convention); ≥1 negative control per detection direction; case set = normal (coupled+unstamped flagged) + error (usage → 4) + boundary (empty dir → 0; Sources-only mention → not flagged; stamped-current → not flagged; stamped-old → flagged; --current/env override) | wiki/testing/quality/tests-that-cannot-fail.md | Inline heredoc fixtures — diverges from the established fixture convention; corpus-dependent tests over live wiki/ — breaks on every ingest | bats tests/wiki-lint-model-era.bats — each check has a fixture that must redden it (seeded fault) | +| D8 | Schema documentation surface | AGENTS.md `### Frontmatter` block gains the `verified_model` line + one meaning sentence ("optional: the model generation the page's guidance was verified against; model-coupled pages missing it are surfaced by lint as re-verification candidates"); templates/page.md frontmatter gains the same line with a placeholder comment; wiki-lint SKILL.md is NOT the schema authority (AGENTS.md is the schema layer) | [no-wiki] | Documenting only in SKILL.md — schema truth would live in a workflow file; a new REQUIRED key — retroactively invalidates 271 pages against R6 | grep for `verified_model` in AGENTS.md + templates/page.md; structure-checks still 0 findings on live corpus | + +Note on D6 arithmetic: weights are error 3×4 checks + warn 2×5 checks + info 1×3 checks = 12+10+3 = **25**; `total_weight = 25`. + +`[no-wiki]` decisions (D2 wiki-schema field design, D4 override precedence, D8 schema-layer authority) are ingest candidates — genuinely uncovered conventions per the reviewer. + +## Review + +plan-reviewer subagent, call 1 of 1 (2026-09-03): + +VERDICT: PASS +FINDINGS (all non-blocking): +- R1–R6 each map to at least one Decision row (R1→D2/D8, R2→D1/D3/D5, R3→D4, R4→D6, R5→D7, R6→D5); no uncovered rule. +- All five non-[no-wiki] Wiki basis paths exist under the wiki root and substantively support their decisions. +- [no-wiki] tags verified as genuinely uncovered conventions, not grounding dodges. +- Verified against the live repo: REQUIRED_KEYS has no unknown-key check (D2/R1 holds); exit contract 0/3/4 matches structure-checks; total_weight 24→25 arithmetic correct; test.yml blocking step confirmed limited to the two existing scripts. +- No Decision contradicts analysis.md Constraints; every Rejected alternative concretely justified (D3 backed by measured 3/271 vs 27/271 grep). +SUMMARY: All requirements covered, groundings resolve, constraints hold, baseline reproduced green. PASS. diff --git a/plans/wiki-model-era/plan.md b/plans/wiki-model-era/plan.md new file mode 100644 index 0000000..9e8c759 --- /dev/null +++ b/plans/wiki-model-era/plan.md @@ -0,0 +1,33 @@ +# wiki-model-era + +Goal: The wiki gains an aging-management signal for model-generation debt: an +optional `verified_model` frontmatter field records the model a page's guidance +was verified against, and a mechanical checker reports model-coupled pages whose +field is absent or outdated as re-verification candidates in the wiki-lint +report. Acceptance: all six R-rules in analysis.md hold; live corpus unedited; +CI stays green. + +Stack: Node ≥18 (repo scripts convention, no deps), bats (tests convention), +Markdown/YAML frontmatter (wiki schema). + +## Decisions +| # | Decision | Choice | Wiki basis | +|---|----------|--------|------------| +| D1 | Mechanical/semantic split | Countable half in `scripts/wiki-lint-model-era.js`; semantic half in wiki-lint SKILL.md fix protocol | wiki/qa/process/llm-review-pipelines.md | +| D2 | Frontmatter field | Optional `verified_model: ` after `last_verified`; never a REQUIRED key | [no-wiki] | +| D3 | Model-coupled detection | Case-insensitive keyword set (claude, sonnet, opus, gpt-, llm, subagent, hallucinat, context window), leading boundary `(?` > env `DEV_LOOP_CURRENT_MODELS` > default `opus-4,fable-5`; current = any token is a case-insensitive substring of the field value | [no-wiki] | +| D5 | Output/exit contract + CI | Exit 0 clean / 3 candidates / 4 usage; stdout `pages: N, model-coupled: M, candidates: K`; stderr `revalidate:: `; NOT in test.yml blocking step | wiki/testing/quality/checks-that-cannot-pass.md | +| D6 | SKILL.md integration | Check #12 (info), Phase 0 fourth command, `total_weight = 25`, score pins in tests/wiki-lint-score.bats bumped same task | wiki/qa/document-verification/editing-a-gated-document.md | +| D7 | Test design | Fixtures under tests/fixtures/model-era/ (mini-root per scenario); deciding assertion last per @test; ≥1 negative control per direction | wiki/testing/quality/tests-that-cannot-fail.md | +| D8 | Schema docs | AGENTS.md frontmatter block + meaning sentence; templates/page.md YAML comment line; SKILL.md is not schema authority | [no-wiki] | + +## Size verdict +size: small + +## Task order +| Task | Depends on | Parallel-ok | +|------|-----------|-------------| +| 01-model-era-script | — | parallel-ok with 02 | +| 02-schema-docs | — | parallel-ok with 01 | +| 03-skill-check-row | 01 | — | diff --git a/plans/wiki-model-era/review-verdict.md b/plans/wiki-model-era/review-verdict.md new file mode 100644 index 0000000..5a904ff --- /dev/null +++ b/plans/wiki-model-era/review-verdict.md @@ -0,0 +1 @@ +VERDICT: PASS diff --git a/plans/wiki-model-era/tasks/01-model-era-script.md b/plans/wiki-model-era/tasks/01-model-era-script.md new file mode 100644 index 0000000..3af01e2 --- /dev/null +++ b/plans/wiki-model-era/tasks/01-model-era-script.md @@ -0,0 +1,110 @@ +# Task 01: Model-era candidate checker script with bats tests + +## Objective +`scripts/wiki-lint-model-era.js` exists and mechanically reports model-coupled +wiki pages whose `verified_model` frontmatter is absent or outside the current +model generation, with a bats suite proving every detection direction. + +## Wiki pages (read these first, only these) +- wiki/testing/quality/checks-that-cannot-pass.md — use for: the exit-code + contract (distinct code per outcome; run the check against known-good input + before adopting; predict failure modes first) +- wiki/testing/quality/guard-shape-vs-consequence.md — use for: detector scope + narrowing (exclude by location, not by allowlist) and the seeded-fault rule + (every check needs a fixture that must redden it) +- wiki/testing/quality/tests-that-cannot-fail.md — use for: bats assertion + discipline (deciding assertion is the test's final command; mutation-check + each assertion's ability to fail) +- wiki/testing/quality/minimum-case-set.md — use for: choosing the case set + (normal + error + boundary per input dimension) + +## Inputs +- scripts/wiki-structure-checks.js — the convention source: header comment + style, walk(), frontmatter regex `^---\n([\s\S]*?)\n---`, report(), + exit 0/3/4, summary-on-stdout/findings-on-stderr. +- tests/wiki-lint-prohibitions.bats + tests/fixtures/prohibitions/ — bats and + fixture conventions (issue #114 header comment: deciding assertion last). +- Decisions that bind you: D3 (keyword set + scope), D4 (precedence + substring + match + default `opus-4,fable-5`), D5 (output/exit contract), D7 (fixture + layout). + +## Steps +1. Write `scripts/wiki-lint-model-era.js` (no dependencies, Node ≥18): + - Usage: `node scripts/wiki-lint-model-era.js [--current ]`. + No root, unknown flag, `--current` without value, or unreadable root → + usage/reason on stderr, exit 4. + - Current set: `--current` csv if given, else env `DEV_LOOP_CURRENT_MODELS` + csv if non-empty, else `DEFAULT_CURRENT = ['opus-4', 'fable-5']` (one + documented constant at the top; header comment says why it is overridable). + - Walk `` recursively for `*.md`, skipping files named `index.md`. + - Per page: frontmatter = the leading `^---\n([\s\S]*?)\n---` block if + present; `verified_model` = trimmed value of `/^verified_model:\s*(.*)$/m` + within it, else absent. Body = text after the frontmatter block (whole + file when no frontmatter), minus the `## Sources` section (from + `/^##\s+Sources\b/m` to the next `/^## /m` or EOF), minus markdown link + targets (strip `/\(https?:[^)]*\)/g`). + - Model-coupled = body matches + `/(?' not in current set`. + - Output: stdout exactly one line + `pages: N, model-coupled: M, candidates: K`; stderr one line per candidate + `revalidate:: `; exit 3 if K>0 else 0. +2. Create fixtures, one mini-root directory per scenario (the bats test points + the script at each subdirectory): + - `tests/fixtures/model-era/coupled-unstamped/page.md` — minimal frontmatter + (id/domain/category only is fine — this script reads only + `verified_model`), body sentence mentioning Claude; plus an `index.md` in + the same dir whose body says "claude" (proves index exclusion). + - `tests/fixtures/model-era/coupled-stamped-current/page.md` — same body, + frontmatter has `verified_model: claude-fable-5`. + - `tests/fixtures/model-era/coupled-stamped-old/page.md` — same body, + `verified_model: claude-sonnet-3-7`. + - `tests/fixtures/model-era/uncoupled/page.md` — body with no keyword. + - `tests/fixtures/model-era/sources-only/page.md` — keyword appears only + inside `## Sources` (e.g. an anthropic.com/claude URL), body otherwise + uncoupled. + - `tests/fixtures/model-era/hyphen-id/page.md` — no frontmatter; body + mentions `backend-common-llm-context-window-budget` only (hyphen-prefixed, + must NOT count as coupled). +3. Write `tests/wiki-lint-model-era.bats` (header comment carries the issue + #114 deciding-assertion-last rule, per the sibling suites). Cases, each + against its fixture mini-root: + - normal: coupled-unstamped → exit 3, stdout `candidates: 1`, stderr has + `revalidate:` and `no verified_model`; index.md not listed. + - stamped-current → exit 0, stdout `candidates: 0`. + - stamped-old, default set → exit 3, stderr has `not in current set`. + - stamped-old, `--current sonnet-3` → exit 0. + - stamped-old, env `DEV_LOOP_CURRENT_MODELS=sonnet-3` → exit 0. + - precedence: env `DEV_LOOP_CURRENT_MODELS=sonnet-3` AND + `--current fable-5` → exit 3 (CLI wins). + - uncoupled → exit 0, stdout `model-coupled: 0`. + - sources-only → exit 0, stdout `model-coupled: 0`. + - hyphen-id → exit 0, stdout `model-coupled: 0`. + - boundary: empty directory (`mktemp -d` under `$BATS_TEST_TMPDIR`) → + exit 0, `pages: 0`. + - error: no args → exit 4; nonexistent root → exit 4; `--current` with no + value → exit 4. + - `node --check scripts/wiki-lint-model-era.js` exits 0. +4. Mutation-check per tests-that-cannot-fail: temporarily flip the coupled + regex (e.g. require `zzz`) and require the normal case to redden, then + restore (work is uncommitted — restore by editing back, not + `git checkout --`). + +## Deliverables +- scripts/wiki-lint-model-era.js (new) +- tests/wiki-lint-model-era.bats (new) +- tests/fixtures/model-era/ (new; the six mini-roots above) + +## Verify +- `bats tests/wiki-lint-model-era.bats` → rc=0, all cases pass. +- `node scripts/wiki-lint-model-era.js wiki` on the repo corpus → exit 3, a + summary of shape `pages: 2xx, model-coupled: ~27, candidates: ~27` (record + the actual numbers; they are report data, not a gate). +- covers: R2, R3, R5 +## Out of scope +- Any edit to skills/wiki-lint/SKILL.md, tests/wiki-lint-score.bats, or + .github/workflows/test.yml (task 03 wires the report; CI stays untouched). +- Any edit to wiki/ pages, AGENTS.md, templates/page.md (task 02). diff --git a/plans/wiki-model-era/tasks/02-schema-docs.md b/plans/wiki-model-era/tasks/02-schema-docs.md new file mode 100644 index 0000000..e37ea61 --- /dev/null +++ b/plans/wiki-model-era/tasks/02-schema-docs.md @@ -0,0 +1,53 @@ +# Task 02: Document the verified_model field in the schema layer + +## Objective +AGENTS.md's frontmatter schema and templates/page.md both document the optional +`verified_model` field, and the live corpus still passes structure checks +unchanged. + +## Wiki pages (read these first, only these) +- wiki/qa/document-verification/editing-a-gated-document.md — use for: + inventorying grep/test anchors on AGENTS.md and templates/page.md BEFORE + editing, and re-running the gate set after. + +## Inputs +- AGENTS.md — the `### Frontmatter` section (yaml block around lines 116-134 + at HEAD 300e4be) and the `confidence` meanings paragraph after it. +- templates/page.md — the frontmatter block (lines 1-11). +- Decisions that bind you: D2 (field name/format/optionality), D8 (AGENTS.md is + the schema authority; template carries a YAML comment line). + +## Steps +1. Anchor inventory (record the outputs): `grep -rn "last_verified" tests/` + and `grep -rln "templates/page.md\|### Frontmatter" tests/ hooks/ scripts/` + — confirm no test pins the yaml block's exact line count or content; if a + pin exists, STOP and report BLOCKED (plan defect) rather than editing + around it. +2. In AGENTS.md's `### Frontmatter` yaml block, add after the + `last_verified: YYYY-MM-DD` line: + `verified_model: # optional` +3. After the `confidence` meanings paragraph, add one sentence: + "`verified_model` (optional) names the model generation the page's guidance + was verified against (e.g. `claude-fable-5`); model-coupled pages missing + it, or carrying an outdated one, are surfaced by lint as re-verification + candidates — see `scripts/wiki-lint-model-era.js`." +4. In templates/page.md, add after the `last_verified: YYYY-MM-DD` line: + `# optional: verified_model: ` +5. Re-run the gates the inventory found plus the structure baseline. + +## Deliverables +- AGENTS.md (modified: frontmatter block + one sentence) +- templates/page.md (modified: one comment line) + +## Verify +- `grep -c verified_model AGENTS.md` → 2; `grep -c verified_model templates/page.md` → 1. +- `node scripts/wiki-structure-checks.js wiki` → exit 0 (live corpus + untouched, no new required key enforced). +- `bats tests/wiki-agent-gate.bats` → rc=0 (nearest AGENTS.md-adjacent suite + still green). +- covers: R1 +## Out of scope +- Editing any wiki/ page to add the field (retroactive stamping is explicitly + out of the feature's scope). +- scripts/wiki-structure-checks.js REQUIRED_KEYS (must stay 8 keys). +- skills/wiki-lint/SKILL.md (task 03). diff --git a/plans/wiki-model-era/tasks/03-skill-check-row.md b/plans/wiki-model-era/tasks/03-skill-check-row.md new file mode 100644 index 0000000..1a2ee4d --- /dev/null +++ b/plans/wiki-model-era/tasks/03-skill-check-row.md @@ -0,0 +1,57 @@ +# Task 03: Wire the model-era check into wiki-lint's report + score + +## Objective +skills/wiki-lint/SKILL.md carries the new re-verification-candidate check +(info severity, check #12), its Phase 0 baseline command, a consistent health +score (`total_weight = 25`), and a fix-protocol rule; the prose pins in +tests/wiki-lint-score.bats are bumped to match and the whole suite is green. + +## Wiki pages (read these first, only these) +- wiki/qa/document-verification/editing-a-gated-document.md — use for: anchor + inventory on SKILL.md before editing; re-run the pinning suite after and + compare pass counts. +- wiki/testing/quality/checks-that-cannot-pass.md — use for: keeping the check + row's wording aligned with the script's real exit contract (report-only, + distinct exit codes). + +## Inputs +- scripts/wiki-lint-model-era.js — from task 01 (the row must name this exact + path and its real behavior). +- skills/wiki-lint/SKILL.md — Phase 0, Checks table (rows 1-11), Health score + section, Fix protocol. +- tests/wiki-lint-score.bats — the 5 prose-pin tests. +- Decisions that bind you: D5 (report-only, not CI-blocking), D6 (info + severity, #12, total_weight 25, pins bumped same task). + +## Steps +1. Anchor inventory: `grep -n "total_weight\|three read-only\|10–11\|10-11" tests/wiki-lint-score.bats skills/wiki-lint/SKILL.md` — record hits. +2. SKILL.md Phase 0: add a fourth baseline command bullet: + `node scripts/wiki-lint-model-era.js wiki` (re-verification candidates). +3. SKILL.md Checks table: add row + `| 12 | Model-coupled page (body references model/LLM behavior) whose \`verified_model\` is absent or outside the current generation — re-verification candidate, report-only; detected by \`node scripts/wiki-lint-model-era.js\` (override the current set with \`--current\` or \`DEV_LOOP_CURRENT_MODELS\`) | info |` +4. Health score section: info row checks become `10–12`; replace + `total_weight = 24` with `total_weight = 25` and the formula + `(4×3 + 5×2 + 2×1)` with `(4×3 + 5×2 + 3×1)`. +5. Fix protocol: add + `- For 12: report-only — never stamp \`verified_model\` without actually re-verifying the page's directives against a current-generation model; after re-verifying, update \`verified_model\` + \`last_verified\` together, or rewrite/retire the guidance that no longer applies.` +6. tests/wiki-lint-score.bats: bump the `total_weight = 24` assertion to + `total_weight = 25`; extend the Phase 0 test's final compound with + `&& [[ "$section" == *"wiki-lint-model-era.js"* ]]` and update its @test + name from "three" to "four" command families. +7. Re-run the pinning suite and compare pass counts with the pre-edit run + (5 tests before, 5 after — no silently skipped test). + +## Deliverables +- skills/wiki-lint/SKILL.md (modified) +- tests/wiki-lint-score.bats (modified — the declared pin bump) + +## Verify +- `bats tests/wiki-lint-score.bats` → rc=0 (5 tests). +- `bats tests/wiki-lint-model-era.bats tests/wiki-lint-prohibitions.bats tests/wiki-structure-checks.bats` → rc=0. +- `git diff --stat .github/ wiki/` → empty (CI wiring and corpus untouched). +- covers: R4, R6 +## Out of scope +- Adding the script to .github/workflows/test.yml's blocking step (permanent + red by design — the check is report-only). +- Editing the script itself (task 01 owns it). +- Retroactively stamping any wiki page. diff --git a/scripts/wiki-lint-model-era.js b/scripts/wiki-lint-model-era.js new file mode 100644 index 0000000..7c1689b --- /dev/null +++ b/scripts/wiki-lint-model-era.js @@ -0,0 +1,110 @@ +#!/usr/bin/env node +// wiki-lint check 12 — model-era re-verification candidates (report-only). +// +// A model-coupled page (its body references model/LLM behavior) ages with model +// generations: guidance verified against one generation may be obsolete quirk +// workaround by the next. This script mechanically surfaces the candidates; the +// semantic judgment (re-verify, rewrite, or retire) stays with the wiki-lint +// skill. Same countable/semantic split as scripts/wiki-lint-prohibitions.js +// (check 2) and scripts/wiki-structure-checks.js (checks 1, 3, 4). +// +// COUPLED : body matches a model-subject keyword (leading boundary excludes +// hyphenated page-id mentions like backend-common-llm-*), scanned +// OUTSIDE frontmatter, the `## Sources` section, and markdown link +// URLs; `index.md` files are routing, never scanned. +// CANDIDATE : coupled AND (`verified_model` frontmatter absent, OR its value +// contains no current-generation token as a substring). +// CURRENT : --current beats DEV_LOOP_CURRENT_MODELS (csv) beats +// DEFAULT_CURRENT. The default is deliberately overridable so the +// checker's own list can age without editing callers. +// +// usage: node scripts/wiki-lint-model-era.js [--current ] +// +// exit 0 no candidates — stdout `pages: N, model-coupled: M, candidates: 0` +// exit 3 candidates — same summary on stdout, one per line on stderr: +// `revalidate:: ` +// exit 4 usage / unreadable root — reason on stderr, nothing on stdout +// +// This is report-only by design: the live corpus legitimately carries +// candidates (~27 pages measured 2026-09-03), so it must NOT join the blocking +// wiki-checks step in .github/workflows/test.yml. +'use strict'; + +const fs = require('fs'); +const path = require('path'); + +const DEFAULT_CURRENT = ['opus-4', 'fable-5']; +const COUPLED = /(? [--current ]\n`); + process.exit(4); +} + +let root = null; +let currentArg = null; +const argv = process.argv.slice(2); +for (let i = 0; i < argv.length; i++) { + if (argv[i] === '--current') { + if (i + 1 >= argv.length) usage('--current requires a value'); + currentArg = argv[++i]; + } else if (argv[i].startsWith('--')) { + usage(`unknown flag '${argv[i]}'`); + } else if (root === null) { + root = argv[i]; + } else { + usage(`unexpected argument '${argv[i]}'`); + } +} +if (!root) usage('missing '); +let rootStat; +try { rootStat = fs.statSync(root); } catch { rootStat = null; } +if (!rootStat || !rootStat.isDirectory()) usage(`'${root}' is not a readable directory`); + +const fromCli = currentArg !== null; +const currentCsv = fromCli ? currentArg : (process.env.DEV_LOOP_CURRENT_MODELS || ''); +const current = (fromCli || currentCsv ? currentCsv.split(',') : DEFAULT_CURRENT) + .map((s) => s.trim().toLowerCase()).filter(Boolean); +if (current.length === 0) usage('current-model set is empty'); + +function walk(dir, out = []) { + for (const entry of fs.readdirSync(dir, { withFileTypes: true })) { + const p = path.join(dir, entry.name); + if (entry.isDirectory()) walk(p, out); + else if (entry.name.endsWith('.md') && entry.name !== 'index.md') out.push(p); + } + return out; +} + +const pages = walk(root); +let coupledCount = 0; +const candidates = []; + +for (const p of pages) { + const text = fs.readFileSync(p, 'utf8'); + const m = text.match(/^---\n([\s\S]*?)\n---/); + const fm = m ? m[1] : ''; + let body = m ? text.slice(m[0].length) : text; + // scan scope: drop the `## Sources` section (citation text is not guidance) + // and markdown link URLs (a docs URL naming a model is not model coupling). + body = body.replace(/^##\s+Sources\b[\s\S]*?(?=^## |(?![\s\S]))/m, ''); + body = body.replace(/\(https?:[^)]*\)/g, '()'); + + if (!COUPLED.test(body)) continue; + coupledCount++; + + const vm = fm.match(/^verified_model:\s*(.*)$/m); + const value = vm ? vm[1].trim() : ''; + if (!value) { + candidates.push(`revalidate:${p}: model-coupled, no verified_model`); + } else if (!current.some((tok) => value.toLowerCase().includes(tok))) { + candidates.push(`revalidate:${p}: verified_model '${value}' not in current set`); + } +} + +process.stdout.write(`pages: ${pages.length}, model-coupled: ${coupledCount}, candidates: ${candidates.length}\n`); +if (candidates.length > 0) { + for (const c of candidates) process.stderr.write(c + '\n'); + process.exit(3); +} +process.exit(0); diff --git a/skills/wiki-lint/SKILL.md b/skills/wiki-lint/SKILL.md index a6d597c..a2960dd 100644 --- a/skills/wiki-lint/SKILL.md +++ b/skills/wiki-lint/SKILL.md @@ -2,7 +2,7 @@ name: wiki-lint effort: medium argument-hint: "[optional: changed pages]" -description: Health-check the bundled wiki. Detect unsourced claims, bare prohibitions, broken links, index and page trigger mismatches, vague qualifiers, oversized pages, and stale dates, then fix them; reports a numeric health score (0-100). Use to keep the wiki healthy before drift compounds. +description: Health-check the bundled wiki. Detect unsourced claims, bare prohibitions, broken links, index and page trigger mismatches, vague qualifiers, oversized pages, stale dates, and model-era re-verification candidates, then fix them; reports a numeric health score (0-100). Use to keep the wiki healthy before drift compounds. --- # Lint @@ -23,6 +23,8 @@ output in the report header: `index.md` files) - Checker baselines: `node scripts/wiki-lint-prohibitions.js wiki` and `node scripts/wiki-structure-checks.js wiki` +- Model-era candidates: `node scripts/wiki-lint-model-era.js wiki` (report-only; + exit 3 with candidates is the normal live-corpus state, not a failure) - Recent history: `tail -5 log.md` An assessment produced without the Phase 0 output pasted in its header is non-compliant. @@ -44,6 +46,7 @@ Run all of these; report findings grouped by severity. | 9 | `contradiction` entries in `log.md` still unresolved | warn | | 10 | `gap` entries in `log.md` with no page created after 30 days | info | | 11 | Bare 2-word prohibition cell (e.g. `Never read`) — undecidable by shape between a state value and a real directive, so it is surfaced rather than judged; reported by `node scripts/wiki-lint-prohibitions.js` | info | +| 12 | Model-coupled page (body references model/LLM behavior) whose `verified_model` frontmatter is absent or outside the current model generation — a re-verification candidate, report-only; detected by `node scripts/wiki-lint-model-era.js` (override the current set with `--current ` or `DEV_LOOP_CURRENT_MODELS`) | info | ## Health score @@ -53,9 +56,9 @@ After running all checks, compute `score = round(100 × passed_weight / total_we |----------|--------|--------| | error | 3 | 1–4 | | warn | 2 | 5–9 | -| info | 1 | 10–11 | +| info | 1 | 10–12 | -`total_weight = 24` (4×3 + 5×2 + 2×1). Report `health: NN/100 (errors E, warns W, infos I)` at the top of the report. This score never gates — no exit-code change, no blocking threshold; it exists only so two runs are comparable. +`total_weight = 25` (4×3 + 5×2 + 3×1). Report `health: NN/100 (errors E, warns W, infos I)` at the top of the report. This score never gates — no exit-code change, no blocking threshold; it exists only so two runs are comparable. ## Fix protocol @@ -67,5 +70,9 @@ After running all checks, compute `score = round(100 × passed_weight / total_we one. Do not invent a replacement — report it if none is known with certainty. - For 5: rewrite the sentence as a conditional ("When X, do A") only when the condition is stated elsewhere in the page; otherwise report it. +- For 12: report-only — never stamp `verified_model` without actually + re-verifying the page's directives against a current-generation model; after + re-verifying, update `verified_model` + `last_verified` together, or + rewrite/retire the guidance that no longer applies. - Append `## [YYYY-MM-DD] lint | errors fixed, reported | health NN/100` to `log.md`. - End the report with up to 3 suggested research questions from recurring gaps. diff --git a/templates/page.md b/templates/page.md index c01e819..dfe6c41 100644 --- a/templates/page.md +++ b/templates/page.md @@ -7,6 +7,7 @@ confidence: unverified sources: - last_verified: YYYY-MM-DD +# optional: verified_model: related: [] --- diff --git a/tests/fixtures/model-era/coupled-stamped-current/page.md b/tests/fixtures/model-era/coupled-stamped-current/page.md new file mode 100644 index 0000000..eaa2f6f --- /dev/null +++ b/tests/fixtures/model-era/coupled-stamped-current/page.md @@ -0,0 +1,16 @@ +--- +id: fixture-coupled-stamped-current +domain: fixture +category: model-era +verified_model: claude-fable-5 +--- + +# A model-coupled page verified on the current generation + +## When this applies + +You are prompting Claude to summarize a diff and it truncates long output. + +## Do this + +When the summary truncates, split the diff before sending it. diff --git a/tests/fixtures/model-era/coupled-stamped-old/page.md b/tests/fixtures/model-era/coupled-stamped-old/page.md new file mode 100644 index 0000000..38e24e7 --- /dev/null +++ b/tests/fixtures/model-era/coupled-stamped-old/page.md @@ -0,0 +1,16 @@ +--- +id: fixture-coupled-stamped-old +domain: fixture +category: model-era +verified_model: claude-sonnet-3-7 +--- + +# A model-coupled page verified on an old generation + +## When this applies + +You are prompting Claude to summarize a diff and it truncates long output. + +## Do this + +When the summary truncates, split the diff before sending it. diff --git a/tests/fixtures/model-era/coupled-unstamped/index.md b/tests/fixtures/model-era/coupled-unstamped/index.md new file mode 100644 index 0000000..e1fe10c --- /dev/null +++ b/tests/fixtures/model-era/coupled-unstamped/index.md @@ -0,0 +1,5 @@ +# Fixture index — mentions claude but must never be scanned + +| Page | Load when | +|------|-----------| +| [page](page.md) | claude behavior fixture | diff --git a/tests/fixtures/model-era/coupled-unstamped/page.md b/tests/fixtures/model-era/coupled-unstamped/page.md new file mode 100644 index 0000000..984b6c2 --- /dev/null +++ b/tests/fixtures/model-era/coupled-unstamped/page.md @@ -0,0 +1,15 @@ +--- +id: fixture-coupled-unstamped +domain: fixture +category: model-era +--- + +# A page that leans on model behavior + +## When this applies + +You are prompting Claude to summarize a diff and it truncates long output. + +## Do this + +When the summary truncates, split the diff before sending it. diff --git a/tests/fixtures/model-era/hyphen-id/page.md b/tests/fixtures/model-era/hyphen-id/page.md new file mode 100644 index 0000000..515feff --- /dev/null +++ b/tests/fixtures/model-era/hyphen-id/page.md @@ -0,0 +1,4 @@ +# A page that references a hyphenated wiki page id + +See backend-common-llm-context-window-budget for the sizing rule; this page +only routes to it and says nothing further itself. diff --git a/tests/fixtures/model-era/sources-only/page.md b/tests/fixtures/model-era/sources-only/page.md new file mode 100644 index 0000000..21dd2f9 --- /dev/null +++ b/tests/fixtures/model-era/sources-only/page.md @@ -0,0 +1,19 @@ +--- +id: fixture-sources-only +domain: fixture +category: model-era +--- + +# A page whose only model mention is a citation + +## When this applies + +You are choosing the column order of a composite index. + +## Do this + +Put the equality-filtered column before the range-filtered column. + +## Sources + +- https://docs.anthropic.com/claude/reference — cited for an unrelated LLM aside diff --git a/tests/fixtures/model-era/uncoupled/page.md b/tests/fixtures/model-era/uncoupled/page.md new file mode 100644 index 0000000..81d5fd4 --- /dev/null +++ b/tests/fixtures/model-era/uncoupled/page.md @@ -0,0 +1,15 @@ +--- +id: fixture-uncoupled +domain: fixture +category: model-era +--- + +# A page about database indexing only + +## When this applies + +You are choosing the column order of a composite index. + +## Do this + +Put the equality-filtered column before the range-filtered column. diff --git a/tests/wiki-lint-model-era.bats b/tests/wiki-lint-model-era.bats new file mode 100644 index 0000000..d61d97b --- /dev/null +++ b/tests/wiki-lint-model-era.bats @@ -0,0 +1,117 @@ +#!/usr/bin/env bats +# Tests for scripts/wiki-lint-model-era.js — the mechanical half of wiki-lint +# check 12 (model-era re-verification candidates): a model-coupled page whose +# `verified_model` frontmatter is absent or outside the current model +# generation is reported `revalidate:: ` on stderr, summary on +# stdout, exit 0 clean / 3 candidates / 4 usage. +# +# Every @test here puts its deciding assertion as the final command — one +# assertion per test, or an `&&`-chained final compound — because a mid-test +# `[[ ]]` assertion is silently masked on macOS's bundled bash 3.2 (issue +# #114); only the exit status of the test's last command is honored. + +setup() { + REPO_ROOT="${BATS_TEST_DIRNAME}/.." + SCRIPT="${REPO_ROOT}/scripts/wiki-lint-model-era.js" + FX="${REPO_ROOT}/tests/fixtures/model-era" +} + +# --- normal: coupled page without the field is a candidate ------------------ + +@test "coupled+unstamped page: exit 3, counted, reason 'no verified_model'" { + run node "$SCRIPT" "$FX/coupled-unstamped" + [ "$status" -eq 3 ] && [[ "$output" == *"candidates: 1"* ]] && [[ "$output" == *"revalidate:"* ]] && [[ "$output" == *"model-coupled, no verified_model"* ]] +} + +@test "index.md is never scanned: the coupled index.md in the same dir adds no candidate" { + run node "$SCRIPT" "$FX/coupled-unstamped" + [[ "$output" == *"pages: 1, model-coupled: 1, candidates: 1"* ]] && [[ "$output" != *"index.md"* ]] +} + +# --- normal: current-generation stamp clears the page ----------------------- + +@test "coupled page stamped with a current-generation model: exit 0, no candidate" { + run node "$SCRIPT" "$FX/coupled-stamped-current" + [ "$status" -eq 0 ] && [[ "$output" == *"candidates: 0"* ]] +} + +@test "coupled page stamped with an old-generation model: exit 3, reason 'not in current set'" { + run node "$SCRIPT" "$FX/coupled-stamped-old" + [ "$status" -eq 3 ] && [[ "$output" == *"not in current set"* ]] && [[ "$output" == *"claude-sonnet-3-7"* ]] +} + +# --- override: --current flag and env var, CLI wins -------------------------- + +@test "--current makes the old stamp current: exit 0" { + run node "$SCRIPT" "$FX/coupled-stamped-old" --current sonnet-3 + [ "$status" -eq 0 ] && [[ "$output" == *"candidates: 0"* ]] +} + +@test "DEV_LOOP_CURRENT_MODELS env makes the old stamp current: exit 0" { + DEV_LOOP_CURRENT_MODELS=sonnet-3 run node "$SCRIPT" "$FX/coupled-stamped-old" + [ "$status" -eq 0 ] && [[ "$output" == *"candidates: 0"* ]] +} + +@test "precedence: --current beats DEV_LOOP_CURRENT_MODELS" { + DEV_LOOP_CURRENT_MODELS=sonnet-3 run node "$SCRIPT" "$FX/coupled-stamped-old" --current fable-5 + [ "$status" -eq 3 ] && [[ "$output" == *"not in current set"* ]] +} + +# --- negative controls: pages that must NOT be flagged ----------------------- + +@test "page with no model keyword: not coupled, exit 0" { + run node "$SCRIPT" "$FX/uncoupled" + [ "$status" -eq 0 ] && [[ "$output" == *"model-coupled: 0"* ]] +} + +@test "keyword only inside '## Sources': not coupled, exit 0" { + run node "$SCRIPT" "$FX/sources-only" + [ "$status" -eq 0 ] && [[ "$output" == *"model-coupled: 0"* ]] +} + +@test "hyphen-prefixed page-id mention (…-llm-…) does not count as coupled" { + run node "$SCRIPT" "$FX/hyphen-id" + [ "$status" -eq 0 ] && [[ "$output" == *"model-coupled: 0"* ]] +} + +# --- boundary: empty root ---------------------------------------------------- + +@test "empty directory: exit 0, pages: 0" { + mkdir -p "$BATS_TEST_TMPDIR/empty" + run node "$SCRIPT" "$BATS_TEST_TMPDIR/empty" + [ "$status" -eq 0 ] && [[ "$output" == *"pages: 0, model-coupled: 0, candidates: 0"* ]] +} + +# --- error: usage contract --------------------------------------------------- + +@test "no arguments: exit 4 with usage on stderr" { + run node "$SCRIPT" + [ "$status" -eq 4 ] && [[ "$output" == *"usage"* ]] +} + +@test "nonexistent root: exit 4" { + run node "$SCRIPT" "$BATS_TEST_TMPDIR/does-not-exist" + [ "$status" -eq 4 ] +} + +@test "--current without a value: exit 4" { + run node "$SCRIPT" "$FX/uncoupled" --current + [ "$status" -eq 4 ] +} + +@test "--current with an empty value: exit 4, never a silent fallback to the default set" { + run node "$SCRIPT" "$FX/uncoupled" --current "" + [ "$status" -eq 4 ] && [[ "$output" == *"current-model set is empty"* ]] +} + +@test "unknown flag: exit 4" { + run node "$SCRIPT" "$FX/uncoupled" --frobnicate + [ "$status" -eq 4 ] +} + +# --- the script parses ------------------------------------------------------- + +@test "the script is syntactically valid" { + run node --check "$SCRIPT" + [ "$status" -eq 0 ] +} diff --git a/tests/wiki-lint-score.bats b/tests/wiki-lint-score.bats index 672cb87..d50c9fe 100644 --- a/tests/wiki-lint-score.bats +++ b/tests/wiki-lint-score.bats @@ -20,18 +20,18 @@ section_body() { # --- normal: Phase 0 exists, precedes Checks, names the three commands ----- -@test "Phase 0 exists, precedes Checks, and names all three read-only command families" { +@test "Phase 0 exists, precedes Checks, and names all four read-only command families" { p0="$(grep -n '^## Phase 0' "$SKILL" | head -1 | cut -d: -f1)" chk="$(grep -n '^## Checks' "$SKILL" | head -1 | cut -d: -f1)" section="$(section_body "$SKILL" '^## Phase 0')" - [ -n "$p0" ] && [ -n "$chk" ] && [ "$p0" -lt "$chk" ] && [[ "$section" == *"wc -l"* ]] && [[ "$section" == *"wiki-lint-prohibitions.js"* ]] && [[ "$section" == *"wiki-structure-checks.js"* ]] + [ -n "$p0" ] && [ -n "$chk" ] && [ "$p0" -lt "$chk" ] && [[ "$section" == *"wc -l"* ]] && [[ "$section" == *"wiki-lint-prohibitions.js"* ]] && [[ "$section" == *"wiki-structure-checks.js"* ]] && [[ "$section" == *"wiki-lint-model-era.js"* ]] } # --- normal: score formula constants are spelled out ----------------------- -@test "Health score section states total_weight=24, weights 3/2/1, and the health: NN/100 format" { +@test "Health score section states total_weight=25, weights 3/2/1, and the health: NN/100 format" { section="$(section_body "$SKILL" '^## Health score')" - [[ "$section" == *"total_weight = 24"* ]] && [[ "$section" == *"| error | 3 |"* ]] && [[ "$section" == *"| warn | 2 |"* ]] && [[ "$section" == *"| info | 1 |"* ]] && [[ "$section" == *"health: NN/100 (errors E, warns W, infos I)"* ]] + [[ "$section" == *"total_weight = 25"* ]] && [[ "$section" == *"| error | 3 |"* ]] && [[ "$section" == *"| warn | 2 |"* ]] && [[ "$section" == *"| info | 1 |"* ]] && [[ "$section" == *"health: NN/100 (errors E, warns W, infos I)"* ]] } # --- error/negative: the compliance clause must be present verbatim --------