From 429617ea54f8861f1fcd6897458e7d7b9e851d15 Mon Sep 17 00:00:00 2001 From: Scott Pfister Date: Mon, 3 Aug 2026 11:24:28 -0500 Subject: [PATCH 01/11] add concise-technical-writing skill - initial commit --- skills/concise-technical-writing/SKILL.md | 231 ++++++++++++++++++++++ 1 file changed, 231 insertions(+) create mode 100644 skills/concise-technical-writing/SKILL.md diff --git a/skills/concise-technical-writing/SKILL.md b/skills/concise-technical-writing/SKILL.md new file mode 100644 index 0000000..6def750 --- /dev/null +++ b/skills/concise-technical-writing/SKILL.md @@ -0,0 +1,231 @@ +--- +name: concise-technical-writing +description: Use when creating, editing, or refining durable technical communication such as documentation, code comments, PR descriptions, issue bodies, review replies, handoff notes, architecture notes, runbooks, agent instructions, or technical explanations where clarity, concision, structure, and unambiguous wording matter. Apply this skill implicitly for final wording of durable artifacts. +author: + name: Scott Pfister + email: scott.pfister@7factor.io +--- + +# Concise Technical Writing + +Use this skill as a final communication pass for durable technical writing. + +The goal is precision: engineering communication that is clear, concise, explicit, and easy for humans and agents to reuse. + +## Default Behavior + +Use this skill when the output is likely to be saved, reviewed, reused, searched, pasted, committed, or used by another agent later. + +Examples: + +- Code comments +- README sections +- Architecture docs +- ADRs +- API docs +- PR descriptions +- Review replies +- Issue descriptions +- Handoff docs +- Task plans +- Runbooks +- Agent skills +- Project instructions + +For normal conversation, use the principles lightly. Keep replies natural. Apply the full refinement pass when the output is durable or precision matters. + +## Workflow + +Before finalizing durable technical writing: + +1. Identify the artifact type. +2. Identify the dominant communication intent. +3. Select a writing mode. +4. Apply section-level modes when a section has a different intent. +5. Refine for clarity, concision, and structure. +6. Preserve claim strength. Keep assumptions labeled as assumptions. +7. Check that the final text keeps the original meaning. + +## Intent Classifier + +Classify by intent first and artifact type second. + +- `instruct`: Tell someone what to do. +- `specify`: State requirements, contracts, rules, invariants, or acceptance criteria. +- `reference`: Help future lookup. +- `explain`: Help understanding. +- `justify`: Explain rationale, tradeoffs, or risk. +- `explore`: Think through unknowns or options. +- `respond`: Answer a person, especially in review or collaboration. + +## Modes + +### `auto` + +Default mode. Classify the artifact and intent, then choose the right mode. + +Use a dominant mode for the artifact. Override by section only when the section's intent clearly differs. + +### `engineering` + +Use for concise natural technical prose. + +Good for: + +- PR descriptions +- Explainers +- Review replies +- Design summaries +- Normal documentation +- Rationale that does not need a long narrative + +Rules: + +- Prefer short paragraphs. +- Remove filler and generic praise. +- Use specific nouns and verbs. +- Keep terminology consistent. +- State assumptions and limits. +- Separate summary, details, risks, and verification when useful. +- Keep a human tone when replying to people. + +### `controlled` + +Use controlled technical English inspired by ASD-STE100. Use software terminology instead of the official STE approved word list. + +Good for: + +- Procedures +- Code comments +- API docs +- Runbooks +- Acceptance criteria +- Requirements +- Contracts +- Invariants +- Implementation notes + +Rules: + +- Use active voice. +- Use explicit subjects. +- Put one action or claim in each sentence. +- State conditions before actions. +- Use the same term for the same concept. +- Prefer concrete verbs over abstract nouns. +- Remove filler, hedging, and marketing language. +- Keep sentences short when practical. +- Use ordered lists for procedures. +- Use bullets or tables for sets of facts. +- Separate facts, assumptions, recommendations, and rationale. +- Make error states and consequences explicit. + +Use software vocabulary. Keep precise terms even when they are outside aircraft-maintenance vocabulary. Prefer clear sentences over mechanical rule compliance. + +### `reference` + +Use for dense, predictable lookup material. + +Good for: + +- Architecture maps +- Module summaries +- Repo guides +- Handoff state +- Agent-facing memory +- Source indexes + +Optimize for agent retrieval first and human readability second. + +Rules: + +- Prefer stable headings and fields. +- Use sparse prose. +- Use explicit names, paths, commands, owners, states, and links. +- Group facts under predictable labels. +- Include sources when available. +- Do not hide important facts in paragraphs. + +Useful fields: + +- Purpose +- Responsibilities +- Inputs +- Outputs +- Dependencies +- Invariants +- Failure Modes +- Sources +- Verification +- Open Questions + +### `narrative` + +Use when nuance, exploration, persuasion, or historical context matters. + +Good for: + +- Design exploration +- Tradeoff discussion +- Strategy +- RFC discussion +- ADR rationale +- Persuasive review context + +Rules: + +- Keep the prose clear and concise, but allow more connective tissue. +- Preserve uncertainty and disagreement. +- Explain why options were accepted or rejected. +- Do not flatten tradeoffs into false certainty. +- Keep facts separate from opinions and recommendations. + +## Common Artifact Mapping + +- Code comment: usually `controlled`; use `engineering` only for rationale. +- PR description: usually `engineering`; use `controlled` for testing, rollout, and reviewer instructions. +- Review reply: usually `engineering`; use `controlled` for exact commitments or steps. +- Runbook: usually `controlled`; use `engineering` for background. +- Architecture index: usually `reference`; use `engineering` for short context. +- ADR: mixed. Decision and consequences use `controlled`; rationale uses `engineering`; exploration uses `narrative`. +- Handoff doc: usually `reference`; use `controlled` for next steps and commands. +- Agent skill: usually `controlled` for procedure; use `reference` for lookup tables; use `engineering` for short context. + +## Claim Safety + +Concise writing must not overstate certainty. + +- Do not strengthen claims during refinement. +- Preserve uncertainty when the source is uncertain. +- Mark assumptions explicitly. +- Mark inferences explicitly when useful. +- Do not convert guesses into facts. +- Include source paths, commands, or evidence when the artifact is durable and evidence exists. +- If an important claim is unverified, label it as unverified or ask whether to verify it. + +Use clear labels when needed: + +- `Fact:` +- `Assumption:` +- `Inference:` +- `Recommendation:` +- `Unknown:` + +## Embedded Use Contract + +Other skills can depend on this skill with this compact instruction: + +> Before finalizing durable technical writing, apply `concise-technical-writing`: classify intent, select a mode, refine for clarity and concision, preserve claim strength, and optimize structure for later retrieval when applicable. + +## Anti-Patterns + +Avoid: + +- Applying controlled mode to brainstorming or early design exploration. +- Making human replies sound like maintenance procedures. +- Removing useful nuance from rationale. +- Replacing precise software terminology with generic words. +- Hiding assumptions to make the text shorter. +- Turning every artifact into a prose essay. +- Turning every artifact into a rigid template. +- Adding headings when a short answer is enough. From 56eea3e386916dde7b1bd70291a85b3e1aab7343 Mon Sep 17 00:00:00 2001 From: Scott Pfister Date: Mon, 3 Aug 2026 12:51:32 -0500 Subject: [PATCH 02/11] docs: list concise-technical-writing in README catalog The skill was added without a corresponding entry under Available Skills. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/README.md b/README.md index ae77dc8..7b1b5e0 100644 --- a/README.md +++ b/README.md @@ -10,6 +10,9 @@ directory with a `SKILL.md` file that describes when the skill should be loaded - `mentor`: switches an agent into learning-first mentoring mode for developing engineers. - `claude-usage-report`: reports Claude Code usage & cost from local session transcripts. (For per-account attribution across multiple accounts, install as a plugin instead — [see below](#installing-claude-usage-report-skill-vs-plugin).) +- `concise-technical-writing`: picks a writing mode per artifact for durable technical text — docs, code + comments, PR descriptions, runbooks, handoff notes — and holds claim strength steady while tightening wording. + Procedural text follows controlled English modeled on [ASD-STE100](https://www.asd-ste100.org/). ## Install From cea72c476813d2b6a1ac13c4d2e6bc8202f51fca Mon Sep 17 00:00:00 2001 From: Scott Pfister Date: Mon, 3 Aug 2026 13:02:20 -0500 Subject: [PATCH 03/11] refactor(concise-technical-writing): collapse to gears, cut duplication MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 231 lines -> 131. The repo's write-a-skill checklist caps SKILL.md at 100 lines; the remainder is two mapping tables that every run consumes, so disclosing them to a sibling file would cost more than it saves. - Reinstate "gears" as the leading word. The four modes are one style at four compression ratios, so order them on that axis and give the dial a default (start in 2) instead of a per-artifact lookup. - Collapse seven artifact enumerations (58 entries, one taxonomy) into a single intent table and a single artifact fallback table. - Delete Anti-Patterns. Eight prohibitions, six of which restated a positive rule; the two live ones became rows in the artifact table. Steering by prohibition names the behavior it bans. - Flip the remaining 14 "Do not" bullets to positive form. - Drop no-op rules the model already follows by default. - Restore the ~20-word sentence bound, which makes gear 3 checkable. - Rename the `reference` intent to `look-up`; it collided with the `reference` mode. - Give the workflow a checkable completion criterion: every output claim traces to an input claim at equal or weaker strength. - Add "Refining text that already exists" — rebuild from an extracted claim list, since a rewrite keys off the prose it reads. - Attribute ASD-STE100 to non-native readers, not aircraft maintenance. - Note prose-linter enforcement for style drift. - Halve the description; it renamed one branch ten times. Co-Authored-By: Claude Opus 5 (1M context) --- skills/concise-technical-writing/SKILL.md | 270 +++++++--------------- 1 file changed, 85 insertions(+), 185 deletions(-) diff --git a/skills/concise-technical-writing/SKILL.md b/skills/concise-technical-writing/SKILL.md index 6def750..801a8b7 100644 --- a/skills/concise-technical-writing/SKILL.md +++ b/skills/concise-technical-writing/SKILL.md @@ -1,6 +1,6 @@ --- name: concise-technical-writing -description: Use when creating, editing, or refining durable technical communication such as documentation, code comments, PR descriptions, issue bodies, review replies, handoff notes, architecture notes, runbooks, agent instructions, or technical explanations where clarity, concision, structure, and unambiguous wording matter. Apply this skill implicitly for final wording of durable artifacts. +description: Use when writing or refining durable technical text — docs, code comments, PR descriptions, issue bodies, runbooks, handoff notes, agent instructions — or when another skill needs a final wording pass. Applies implicitly to durable artifacts. author: name: Scott Pfister email: scott.pfister@7factor.io @@ -8,224 +8,124 @@ author: # Concise Technical Writing -Use this skill as a final communication pass for durable technical writing. +Write for precision: an engineer or agent reading this later must not have to guess what it meant. -The goal is precision: engineering communication that is clear, concise, explicit, and easy for humans and agents to reuse. +## Gears -## Default Behavior +Control is a dial, not a switch. The four gears are one style at four compression ratios, ordered from most prose to least. -Use this skill when the output is likely to be saved, reviewed, reused, searched, pasted, committed, or used by another agent later. +| Gear | Name | Prose | Shift here when | +| ---- | ------------- | ----------- | -------------------- | +| 1 | `narrative` | Most | Exploration matters | +| 2 | `engineering` | Default | — | +| 3 | `controlled` | Little | Precision matters | +| 4 | `reference` | Almost none | Later lookup matters | -Examples: +Start in gear 2. Shift to 3 or 4 when precision or lookup matters. Drop to gear 1 only when exploration, persuasion, or live disagreement matters. -- Code comments -- README sections -- Architecture docs -- ADRs -- API docs -- PR descriptions -- Review replies -- Issue descriptions -- Handoff docs -- Task plans -- Runbooks -- Agent skills -- Project instructions +Shift per section, not only per document. An ADR runs gear 3 for the decision, gear 2 for the rationale, gear 1 for the discussion. -For normal conversation, use the principles lightly. Keep replies natural. Apply the full refinement pass when the output is durable or precision matters. +In conversation, keep the reply natural and apply the gear's spirit. Full refinement is for durable text. -## Workflow +## Choosing a gear -Before finalizing durable technical writing: +Classify intent first, artifact second. -1. Identify the artifact type. -2. Identify the dominant communication intent. -3. Select a writing mode. -4. Apply section-level modes when a section has a different intent. -5. Refine for clarity, concision, and structure. -6. Preserve claim strength. Keep assumptions labeled as assumptions. -7. Check that the final text keeps the original meaning. +| Intent | Gear | +| --------------------------------------------------------------- | ---- | +| `instruct` — tell someone what to do | 3 | +| `specify` — state requirements, contracts, invariants, criteria | 3 | +| `look-up` — help someone find a fact later | 4 | +| `explain` — help someone understand | 2 | +| `justify` — give rationale, tradeoffs, or risk | 2 | +| `respond` — answer a person, in review or collaboration | 2 | +| `explore` — think through unknowns or options | 1 | -## Intent Classifier +When intent is mixed or unclear, fall back to the artifact: -Classify by intent first and artifact type second. +| Artifact | Gear | Shift for | +| ----------------------------------------------- | ------------------ | -------------------------------------- | +| Code comment | 3 | 2 for rationale | +| API doc, runbook, procedure, acceptance criteria | 3 | 2 for background | +| Agent skill, project instructions | 3 | 4 for lookup tables, 2 for context | +| PR description | 2 | 3 for testing, rollout, reviewer steps | +| Review reply | 2 | 3 for exact commitments | +| Explainer, design summary, issue body | 2 | — | +| Architecture index, module summary, repo guide | 4 | 2 for short context | +| Handoff note | 4 | 3 for next steps and commands | +| ADR | 3 for the decision | 2 for rationale, 1 for discussion | +| Brainstorm, strategy, RFC discussion | 1 | — | -- `instruct`: Tell someone what to do. -- `specify`: State requirements, contracts, rules, invariants, or acceptance criteria. -- `reference`: Help future lookup. -- `explain`: Help understanding. -- `justify`: Explain rationale, tradeoffs, or risk. -- `explore`: Think through unknowns or options. -- `respond`: Answer a person, especially in review or collaboration. +## Gear rules -## Modes +Each gear adds only what is listed here. -### `auto` +### 1 `narrative` -Default mode. Classify the artifact and intent, then choose the right mode. - -Use a dominant mode for the artifact. Override by section only when the section's intent clearly differs. - -### `engineering` - -Use for concise natural technical prose. - -Good for: - -- PR descriptions -- Explainers -- Review replies -- Design summaries -- Normal documentation -- Rationale that does not need a long narrative - -Rules: - -- Prefer short paragraphs. -- Remove filler and generic praise. -- Use specific nouns and verbs. -- Keep terminology consistent. -- State assumptions and limits. -- Separate summary, details, risks, and verification when useful. -- Keep a human tone when replying to people. - -### `controlled` +- Preserve uncertainty and disagreement. +- Say why each option was accepted or rejected. +- Leave tradeoffs as tradeoffs. +- Label facts, opinions, and recommendations separately. -Use controlled technical English inspired by ASD-STE100. Use software terminology instead of the official STE approved word list. +### 2 `engineering` -Good for: +- Give each paragraph one purpose. +- Cut filler, hedging, and marketing language. +- Name the assumptions and the limits. +- Split summary, detail, risk, and verification when the reader needs them apart. -- Procedures -- Code comments -- API docs -- Runbooks -- Acceptance criteria -- Requirements -- Contracts -- Invariants -- Implementation notes +### 3 `controlled` -Rules: +Controlled English modeled on ASD-STE100, with software vocabulary in place of the approved word list. ASD-STE100 exists to remove ambiguity for readers who are not native English speakers. Write for that reader. -- Use active voice. -- Use explicit subjects. -- Put one action or claim in each sentence. -- State conditions before actions. -- Use the same term for the same concept. -- Prefer concrete verbs over abstract nouns. -- Remove filler, hedging, and marketing language. -- Keep sentences short when practical. +- Use active voice and an explicit subject. +- Put one action or one claim in each sentence. +- State the condition before the action. +- Keep sentences under about 20 words. +- Use the same term for the same concept every time. +- Use concrete verbs in place of abstract nouns. - Use ordered lists for procedures. -- Use bullets or tables for sets of facts. -- Separate facts, assumptions, recommendations, and rationale. -- Make error states and consequences explicit. - -Use software vocabulary. Keep precise terms even when they are outside aircraft-maintenance vocabulary. Prefer clear sentences over mechanical rule compliance. - -### `reference` - -Use for dense, predictable lookup material. +- Name each error state and its consequence. +- Keep precise software terms. A clear sentence beats rule compliance. -Good for: +### 4 `reference` -- Architecture maps -- Module summaries -- Repo guides -- Handoff state -- Agent-facing memory -- Source indexes +Built for an agent to retrieve first and a human to read second. -Optimize for agent retrieval first and human readability second. - -Rules: - -- Prefer stable headings and fields. -- Use sparse prose. -- Use explicit names, paths, commands, owners, states, and links. -- Group facts under predictable labels. -- Include sources when available. -- Do not hide important facts in paragraphs. - -Useful fields: - -- Purpose -- Responsibilities -- Inputs -- Outputs -- Dependencies -- Invariants -- Failure Modes -- Sources -- Verification -- Open Questions - -### `narrative` - -Use when nuance, exploration, persuasion, or historical context matters. - -Good for: - -- Design exploration -- Tradeoff discussion -- Strategy -- RFC discussion -- ADR rationale -- Persuasive review context - -Rules: - -- Keep the prose clear and concise, but allow more connective tissue. -- Preserve uncertainty and disagreement. -- Explain why options were accepted or rejected. -- Do not flatten tradeoffs into false certainty. -- Keep facts separate from opinions and recommendations. +- Use stable headings and field names. +- Put facts under predictable labels, where a reader finds them without reading prose. +- Give explicit names, paths, commands, owners, states, and links. +- Use these fields where they apply: Purpose, Responsibilities, Inputs, Outputs, Dependencies, Invariants, Failure Modes, Sources, Verification, Open Questions. -## Common Artifact Mapping +## Claim safety -- Code comment: usually `controlled`; use `engineering` only for rationale. -- PR description: usually `engineering`; use `controlled` for testing, rollout, and reviewer instructions. -- Review reply: usually `engineering`; use `controlled` for exact commitments or steps. -- Runbook: usually `controlled`; use `engineering` for background. -- Architecture index: usually `reference`; use `engineering` for short context. -- ADR: mixed. Decision and consequences use `controlled`; rationale uses `engineering`; exploration uses `narrative`. -- Handoff doc: usually `reference`; use `controlled` for next steps and commands. -- Agent skill: usually `controlled` for procedure; use `reference` for lookup tables; use `engineering` for short context. +Tightening the wording must not tighten the certainty. This rule outranks concision. -## Claim Safety +- Carry each claim across at its original strength. +- Label assumptions as assumptions and unknowns as unknowns. +- Cite the source path, command, or evidence for each claim in durable text. +- Say a claim is unverified, or ask to verify it, rather than writing around it. +- Add nothing the source did not contain: no internal detail, no commitment, no date. -Concise writing must not overstate certainty. +Use labels where the distinction carries weight: `Fact:` `Assumption:` `Unknown:` -- Do not strengthen claims during refinement. -- Preserve uncertainty when the source is uncertain. -- Mark assumptions explicitly. -- Mark inferences explicitly when useful. -- Do not convert guesses into facts. -- Include source paths, commands, or evidence when the artifact is durable and evidence exists. -- If an important claim is unverified, label it as unverified or ask whether to verify it. +Done when every claim in the output traces to a claim in the input at equal or weaker strength. -Use clear labels when needed: +## Refining text that already exists -- `Fact:` -- `Assumption:` -- `Inference:` -- `Recommendation:` -- `Unknown:` +A rewrite keys off the prose it reads, so vocabulary changes and weak structure survives. Rebuild instead: -## Embedded Use Contract +1. Extract the claims, steps, and open questions as a bare list. +2. Pick the gear from that list, not from the old prose. +3. Write from the list. +4. Check the claim-safety criterion against the original. -Other skills can depend on this skill with this compact instruction: +## Embedded use contract -> Before finalizing durable technical writing, apply `concise-technical-writing`: classify intent, select a mode, refine for clarity and concision, preserve claim strength, and optimize structure for later retrieval when applicable. +Other skills reach this skill with: -## Anti-Patterns +> Before finalizing durable technical writing, apply `concise-technical-writing`: pick a gear, write from claims, hold claim strength steady, and structure for later retrieval. -Avoid: +## Drift -- Applying controlled mode to brainstorming or early design exploration. -- Making human replies sound like maintenance procedures. -- Removing useful nuance from rationale. -- Replacing precise software terminology with generic words. -- Hiding assumptions to make the text shorter. -- Turning every artifact into a prose essay. -- Turning every artifact into a rigid template. -- Adding headings when a short answer is enough. +A gear holds for a few turns, then slips. Where a repo needs the style enforced instead of requested, gate on a prose linter such as [Vale](https://vale.sh) at pre-commit or `PostToolUse`. The skill sets the target; the gate holds it. From 65cf5b0196296a500e10ef8ed98bba86f6dbd90e Mon Sep 17 00:00:00 2001 From: Scott Pfister Date: Mon, 3 Aug 2026 13:24:48 -0500 Subject: [PATCH 04/11] refactor: rename to precise-technical-writing; make Sources load-bearing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Concision is instrumental to precision, not the goal. ASD-STE100 rule 4.2 forbids shortening by omission, and the skill's own claim-safety rule already outranks concision — so the old name pointed at the property the skill subordinates. Gear 4 now requires a Sources field instead of listing it as optional. A reference doc describes something it is not connected to; the path is what lets a reader re-check it. Dropped the Verification field: re-checking a doc against code is a separate concern from wording it. Frontmatter: author is not in the agentskills.io specification, so move it under metadata as a single string. Co-Authored-By: Claude Opus 5 (1M context) --- README.md | 2 +- skills/claude-usage-report/KNOWN-ISSUES.md | 30 +++++++++++++++++++ .../SKILL.md | 14 ++++----- 3 files changed, 38 insertions(+), 8 deletions(-) create mode 100644 skills/claude-usage-report/KNOWN-ISSUES.md rename skills/{concise-technical-writing => precise-technical-writing}/SKILL.md (92%) diff --git a/README.md b/README.md index 7b1b5e0..ae6634a 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ directory with a `SKILL.md` file that describes when the skill should be loaded - `mentor`: switches an agent into learning-first mentoring mode for developing engineers. - `claude-usage-report`: reports Claude Code usage & cost from local session transcripts. (For per-account attribution across multiple accounts, install as a plugin instead — [see below](#installing-claude-usage-report-skill-vs-plugin).) -- `concise-technical-writing`: picks a writing mode per artifact for durable technical text — docs, code +- `precise-technical-writing`: picks a writing mode per artifact for durable technical text — docs, code comments, PR descriptions, runbooks, handoff notes — and holds claim strength steady while tightening wording. Procedural text follows controlled English modeled on [ASD-STE100](https://www.asd-ste100.org/). diff --git a/skills/claude-usage-report/KNOWN-ISSUES.md b/skills/claude-usage-report/KNOWN-ISSUES.md new file mode 100644 index 0000000..f0885fe --- /dev/null +++ b/skills/claude-usage-report/KNOWN-ISSUES.md @@ -0,0 +1,30 @@ +# Known issues / backlog + +## Day-bucketing uses UTC, not local timezone — undercounts late-evening sessions + +`usage_report.py`'s BY DAY (and single-day-argument) logic buckets each message by the +UTC date of its timestamp. For a user in a negative-UTC-offset timezone (e.g. CDT, +UTC-5), any session that runs past ~7pm local time rolls into the *next* UTC calendar +date. The report then splits that one real evening session across two "day" buckets — +and if you query a single day (e.g. `usage_report.py 2026-07-20`), you only see the +pre-midnight-UTC half, silently undercounting that day's actual spend. + +**Reproduced 2026-07-21**: querying `2026-07-20` alone showed session `b8c257de` (a CE-31 +Jira/instrumentation review) at $3.27. Widening the query to `2026-07-20..2026-07-21` +showed the same session — which ran 22:09 UTC 07-20 through 02:20 UTC 07-21, i.e. +5:09pm–9:20pm CDT, entirely within local 07-20 — actually cost $9.08. Same story for +session `6cc3db71` ($0.44 vs. $1.70 full). Together that's ~$7 of same-local-day spend +that a single-day query hid. + +**Fix**: bucket by local date instead of UTC. Simplest approach — accept a +`--tz`/config-driven UTC offset (or read the system local timezone) and convert each +message timestamp to local time before taking its date for BY DAY / single-day-argument +filtering. Also consider: the account-attribution and BY SESSION timestamp columns are +UTC-labeled and unambiguous as-is, so those probably don't need to change — only the +date-bucketing/filtering logic does. + +## BY DAY table isn't ordered by date + +`BY DAY` (and the report skeleton's "By day" table) currently prints in whatever order +the day-cost dict iterates, not chronologically. Sort ascending (or descending, pick one +and document it) by date before printing/writing the table. diff --git a/skills/concise-technical-writing/SKILL.md b/skills/precise-technical-writing/SKILL.md similarity index 92% rename from skills/concise-technical-writing/SKILL.md rename to skills/precise-technical-writing/SKILL.md index 801a8b7..d2ae565 100644 --- a/skills/concise-technical-writing/SKILL.md +++ b/skills/precise-technical-writing/SKILL.md @@ -1,12 +1,11 @@ --- -name: concise-technical-writing +name: precise-technical-writing description: Use when writing or refining durable technical text — docs, code comments, PR descriptions, issue bodies, runbooks, handoff notes, agent instructions — or when another skill needs a final wording pass. Applies implicitly to durable artifacts. -author: - name: Scott Pfister - email: scott.pfister@7factor.io +metadata: + author: Scott Pfister (scott.pfister@7factor.io) --- -# Concise Technical Writing +# Precise Technical Writing Write for precision: an engineer or agent reading this later must not have to guess what it meant. @@ -95,7 +94,8 @@ Built for an agent to retrieve first and a human to read second. - Use stable headings and field names. - Put facts under predictable labels, where a reader finds them without reading prose. - Give explicit names, paths, commands, owners, states, and links. -- Use these fields where they apply: Purpose, Responsibilities, Inputs, Outputs, Dependencies, Invariants, Failure Modes, Sources, Verification, Open Questions. +- Use these fields where they apply: Purpose, Responsibilities, Inputs, Outputs, Dependencies, Invariants, Failure Modes, Open Questions. +- Always include a `Sources` field listing the paths the facts came from. A reference doc describes something it is not connected to, so the reader needs the path to re-check it. Without one, the doc drifts and nobody can tell. ## Claim safety @@ -124,7 +124,7 @@ A rewrite keys off the prose it reads, so vocabulary changes and weak structure Other skills reach this skill with: -> Before finalizing durable technical writing, apply `concise-technical-writing`: pick a gear, write from claims, hold claim strength steady, and structure for later retrieval. +> Before finalizing durable technical writing, apply `precise-technical-writing`: pick a gear, write from claims, hold claim strength steady, and structure for later retrieval. ## Drift From 9ca854579efd9c82004582f90f63c158f254f1ef Mon Sep 17 00:00:00 2001 From: Scott Pfister Date: Mon, 3 Aug 2026 13:34:39 -0500 Subject: [PATCH 05/11] test(eval): add technical-writing eval harness MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Makes the Hacker News critique falsifiable instead of arguable. Four arms — no instruction, hsaliak's one-line prompt, Orwell's six rules, the full skill — over six tasks, one per gear. Two tasks carry the experiment. incident-reply reproduces the failure atoav found in the reference skill's own example output: an agent told to simplify language added internal detail and a customer commitment that were not in the source. Its input holds an unconfirmed cause, an internal hostname, and no agreed date. tradeoff tests the opposite error, over-application: its input is unresolved disagreement that controlled language would flatten. Scoring keeps style and fidelity apart and never sums them. Length is reported, never scored: ASD-STE100 rule 4.2 forbids shortening by omission, and the thread's own top exchange shows a two-word cut changing a meaning. Judging is blind — arm labels stripped, order shuffled on a fixed seed. Known confound, documented rather than hidden: claude --bare is the only mode that skips user-memory discovery and it needs ANTHROPIC_API_KEY, so on an OAuth-only machine every arm carries the operator's global CLAUDE.md. Verified by probing each configuration for a string unique to that file. Contamination is constant across arms, so relative ordering holds and absolute numbers do not. Co-Authored-By: Claude Opus 5 (1M context) --- eval/technical-writing/.gitignore | 1 + eval/technical-writing/README.md | 136 +++++++++ eval/technical-writing/arms/baseline.txt | 0 eval/technical-writing/arms/oneliner.txt | 1 + eval/technical-writing/arms/orwell.txt | 8 + eval/technical-writing/arms/skill.txt | 131 +++++++++ eval/technical-writing/judge.py | 123 +++++++++ eval/technical-writing/run.sh | 107 ++++++++ eval/technical-writing/score.py | 257 ++++++++++++++++++ eval/technical-writing/tasks/code-comment.md | 42 +++ .../technical-writing/tasks/incident-reply.md | 53 ++++ .../technical-writing/tasks/pr-description.md | 41 +++ .../tasks/reference-summary.md | 52 ++++ eval/technical-writing/tasks/runbook.md | 44 +++ eval/technical-writing/tasks/tradeoff.md | 57 ++++ 15 files changed, 1053 insertions(+) create mode 100644 eval/technical-writing/.gitignore create mode 100644 eval/technical-writing/README.md create mode 100644 eval/technical-writing/arms/baseline.txt create mode 100644 eval/technical-writing/arms/oneliner.txt create mode 100644 eval/technical-writing/arms/orwell.txt create mode 100644 eval/technical-writing/arms/skill.txt create mode 100644 eval/technical-writing/judge.py create mode 100755 eval/technical-writing/run.sh create mode 100755 eval/technical-writing/score.py create mode 100644 eval/technical-writing/tasks/code-comment.md create mode 100644 eval/technical-writing/tasks/incident-reply.md create mode 100644 eval/technical-writing/tasks/pr-description.md create mode 100644 eval/technical-writing/tasks/reference-summary.md create mode 100644 eval/technical-writing/tasks/runbook.md create mode 100644 eval/technical-writing/tasks/tradeoff.md diff --git a/eval/technical-writing/.gitignore b/eval/technical-writing/.gitignore new file mode 100644 index 0000000..89f9ac0 --- /dev/null +++ b/eval/technical-writing/.gitignore @@ -0,0 +1 @@ +out/ diff --git a/eval/technical-writing/README.md b/eval/technical-writing/README.md new file mode 100644 index 0000000..3217b63 --- /dev/null +++ b/eval/technical-writing/README.md @@ -0,0 +1,136 @@ +# Technical-writing eval + +Compares four ways to ask a model for precise technical prose, on tasks built to +punish the specific failures each approach claims to fix. + +## The question + +Hacker News made a falsifiable claim about skills like `precise-technical-writing` +([thread](https://news.ycombinator.com/item?id=49114639)): + +> STE is part of the training set, so the skill is redundant and only pollutes your +> context window. — `lab14` + +> Seems to be doing too much, a 1 line in the system prompt is all you need. — `hsaliak` + +That is testable. So is the counter-claim: that the skill earns its tokens through +two things a one-liner cannot carry — gear selection and claim safety. + +Three hypotheses: + +- **H1 — Any instruction beats none.** All three instructed arms lower ambiguity and + structural slop against `baseline`. Expected to hold; it is the sanity check. +- **H2 — Only the skill holds claim strength.** On `incident-reply` and `tradeoff`, + arms without claim-safety rules harden hedged claims, invent detail, or flatten + disagreement. This is the differentiator. If it fails, the skill is cruft. +- **H3 — The skill beats the one-liner by more than its token cost.** The skill is + ~1500 tokens against ~25. If the margin is small, `hsaliak` is right. + +An arm winning on brevity alone proves nothing. ASD-STE100 rule 4.2 forbids +shortening by omission, and the thread's own top exchange shows why: `handfuloflight` +tightened "make sure that your AWS credentials are correct" to "ensure AWS +credentials are correct", and `harshreality` pointed out the rewrite changed the +meaning. Length is therefore **reported but never scored**. + +## Arms + +| Arm | Cost | What it is | +| ---------- | --------- | ------------------------------------------------------ | +| `baseline` | 0 tokens | The task, no style instruction | +| `oneliner` | ~25 | `hsaliak`'s system-prompt line, verbatim from HN | +| `orwell` | ~120 | Orwell's six rules, which beat STE in one HN benchmark | +| `skill` | ~1500 | The full `SKILL.md` | + +`orwell` is in because `gillesjacobs` cited a [benchmark](https://youtu.be/uJblcC4lKYw) +where those six rules beat the STE skill on slop indicators at a fraction of the +tokens. Unverified, and cheap to include as a control. + +## Tasks + +Six tasks, each mapping to a gear the skill would select, and each carrying declared +**traps** — specific failures the scorer looks for by name. + +| Task | Gear | Trap it sets | +| ------------------ | ---- | ---------------------------------------------------- | +| `code-comment` | 3 | Abstract nouns, passive voice, restating the code | +| `pr-description` | 2 | Marketing tone, burying the risk | +| `runbook` | 3 | Action before condition, unnamed failure states | +| `reference-summary`| 4 | Prose instead of fields, omitting `Sources` | +| `incident-reply` | 2 | **Claim inflation** — see below | +| `tradeoff` | 1 | **Over-application** — flattening live disagreement | + +The last two carry the experiment. + +`incident-reply` reproduces the failure `atoav` found in the HN skill's own example +output: an agent told to "just simplify the language" added internal detail and a +customer-facing commitment that were nowhere in the source. Its input contains an +**unconfirmed** root cause, an internal service name, and no agreed fix date. An arm +that states the cause as fact, leaks the internal name, or promises a date has failed +in a way no amount of clean prose redeems. + +`tradeoff` tests the opposite error. Its input is genuine exploration with unresolved +disagreement between two engineers. Controlled language applied here destroys the +content. `baseline` and `orwell` have no mechanism to avoid this; `oneliner` actively +pushes into it. Only the skill has a gear that says stay in prose. + +## Running it + +```sh +cd eval/technical-writing +./run.sh # 24 generations: 6 tasks x 4 arms +./run.sh --model opus # default is sonnet +./run.sh --tasks incident-reply # single task +python3 score.py out/ # deterministic metrics -> out/scores.json +python3 score.py out/ --markdown # readable table +``` + +Then judge blind: + +```sh +python3 judge.py out/ --pairs # emits anonymized A/B pairs + judge.md rubric +``` + +`judge.py` strips arm labels and randomizes presentation order, so the judging model +cannot see which arm wrote what. Run the emitted prompts through any model, or a +second `claude -p` call, and paste verdicts back. + +## Known confound, unresolved + +**Every generation carries the operator's global `~/.claude/CLAUDE.md`.** + +`claude --bare` is the only mode that skips user-memory discovery, and it requires +`ANTHROPIC_API_KEY` — OAuth and keychain are never read in bare mode. On an OAuth-only +machine there is no clean-room path. Verified by asking each configuration whether its +context mentioned a string unique to the operator's global memory; every non-bare +configuration answered yes, including with `--system-prompt` and `--tools ""`. + +Consequence: **absolute** numbers are not clean-room and should not be quoted as such. +**Relative** comparisons stay valid, because the contamination is identical across all +four arms within a run. It is a constant, not a variable. + +To get a clean run, set `ANTHROPIC_API_KEY` and pass `--bare`: + +```sh +ANTHROPIC_API_KEY=sk-... ./run.sh --bare +``` + +`run.sh` adds `--bare` only when that variable is set, and records which mode produced +each run in `out/manifest.json`. + +## What this cannot tell you + +- **Whether the skill fires.** This measures output quality once loaded. Invocation + reliability is a separate experiment against the `description`. +- **Whether the style survives.** Every generation is one turn. `boardwaalk`'s drift + complaint — "models drift immediately" — needs a multi-turn design. +- **Anything about a different model.** Arms may reorder across models. Run the model + you actually use. +- **Ambiguity, directly.** The scorer measures proxies for it. Only the blind judge + reads for meaning, and it is a model, not a panel of tired mechanics. + +## Sources + +- `../../skills/precise-technical-writing/SKILL.md` — the arm under test +- `../../chatgpt-share-asd-ste-100-for.md` — where the gears and modes came from +- https://news.ycombinator.com/item?id=49114639 — the critiques being tested +- https://www.asd-ste100.org/ — the standard gear 3 is modeled on diff --git a/eval/technical-writing/arms/baseline.txt b/eval/technical-writing/arms/baseline.txt new file mode 100644 index 0000000..e69de29 diff --git a/eval/technical-writing/arms/oneliner.txt b/eval/technical-writing/arms/oneliner.txt new file mode 100644 index 0000000..610ac05 --- /dev/null +++ b/eval/technical-writing/arms/oneliner.txt @@ -0,0 +1 @@ +Output tokens are precious, be succinct in your responses. Use ASD-STE100 simplified technical english diff --git a/eval/technical-writing/arms/orwell.txt b/eval/technical-writing/arms/orwell.txt new file mode 100644 index 0000000..a7f4c05 --- /dev/null +++ b/eval/technical-writing/arms/orwell.txt @@ -0,0 +1,8 @@ +Follow Orwell's six rules of writing: + +1. Never use a metaphor, simile, or other figure of speech which you are used to seeing in print. +2. Never use a long word where a short one will do. +3. If it is possible to cut a word out, always cut it out. +4. Never use the passive where you can use the active. +5. Never use a foreign phrase, a scientific word, or a jargon word if you can think of an everyday English equivalent. +6. Break any of these rules sooner than say anything outright barbarous. diff --git a/eval/technical-writing/arms/skill.txt b/eval/technical-writing/arms/skill.txt new file mode 100644 index 0000000..d2ae565 --- /dev/null +++ b/eval/technical-writing/arms/skill.txt @@ -0,0 +1,131 @@ +--- +name: precise-technical-writing +description: Use when writing or refining durable technical text — docs, code comments, PR descriptions, issue bodies, runbooks, handoff notes, agent instructions — or when another skill needs a final wording pass. Applies implicitly to durable artifacts. +metadata: + author: Scott Pfister (scott.pfister@7factor.io) +--- + +# Precise Technical Writing + +Write for precision: an engineer or agent reading this later must not have to guess what it meant. + +## Gears + +Control is a dial, not a switch. The four gears are one style at four compression ratios, ordered from most prose to least. + +| Gear | Name | Prose | Shift here when | +| ---- | ------------- | ----------- | -------------------- | +| 1 | `narrative` | Most | Exploration matters | +| 2 | `engineering` | Default | — | +| 3 | `controlled` | Little | Precision matters | +| 4 | `reference` | Almost none | Later lookup matters | + +Start in gear 2. Shift to 3 or 4 when precision or lookup matters. Drop to gear 1 only when exploration, persuasion, or live disagreement matters. + +Shift per section, not only per document. An ADR runs gear 3 for the decision, gear 2 for the rationale, gear 1 for the discussion. + +In conversation, keep the reply natural and apply the gear's spirit. Full refinement is for durable text. + +## Choosing a gear + +Classify intent first, artifact second. + +| Intent | Gear | +| --------------------------------------------------------------- | ---- | +| `instruct` — tell someone what to do | 3 | +| `specify` — state requirements, contracts, invariants, criteria | 3 | +| `look-up` — help someone find a fact later | 4 | +| `explain` — help someone understand | 2 | +| `justify` — give rationale, tradeoffs, or risk | 2 | +| `respond` — answer a person, in review or collaboration | 2 | +| `explore` — think through unknowns or options | 1 | + +When intent is mixed or unclear, fall back to the artifact: + +| Artifact | Gear | Shift for | +| ----------------------------------------------- | ------------------ | -------------------------------------- | +| Code comment | 3 | 2 for rationale | +| API doc, runbook, procedure, acceptance criteria | 3 | 2 for background | +| Agent skill, project instructions | 3 | 4 for lookup tables, 2 for context | +| PR description | 2 | 3 for testing, rollout, reviewer steps | +| Review reply | 2 | 3 for exact commitments | +| Explainer, design summary, issue body | 2 | — | +| Architecture index, module summary, repo guide | 4 | 2 for short context | +| Handoff note | 4 | 3 for next steps and commands | +| ADR | 3 for the decision | 2 for rationale, 1 for discussion | +| Brainstorm, strategy, RFC discussion | 1 | — | + +## Gear rules + +Each gear adds only what is listed here. + +### 1 `narrative` + +- Preserve uncertainty and disagreement. +- Say why each option was accepted or rejected. +- Leave tradeoffs as tradeoffs. +- Label facts, opinions, and recommendations separately. + +### 2 `engineering` + +- Give each paragraph one purpose. +- Cut filler, hedging, and marketing language. +- Name the assumptions and the limits. +- Split summary, detail, risk, and verification when the reader needs them apart. + +### 3 `controlled` + +Controlled English modeled on ASD-STE100, with software vocabulary in place of the approved word list. ASD-STE100 exists to remove ambiguity for readers who are not native English speakers. Write for that reader. + +- Use active voice and an explicit subject. +- Put one action or one claim in each sentence. +- State the condition before the action. +- Keep sentences under about 20 words. +- Use the same term for the same concept every time. +- Use concrete verbs in place of abstract nouns. +- Use ordered lists for procedures. +- Name each error state and its consequence. +- Keep precise software terms. A clear sentence beats rule compliance. + +### 4 `reference` + +Built for an agent to retrieve first and a human to read second. + +- Use stable headings and field names. +- Put facts under predictable labels, where a reader finds them without reading prose. +- Give explicit names, paths, commands, owners, states, and links. +- Use these fields where they apply: Purpose, Responsibilities, Inputs, Outputs, Dependencies, Invariants, Failure Modes, Open Questions. +- Always include a `Sources` field listing the paths the facts came from. A reference doc describes something it is not connected to, so the reader needs the path to re-check it. Without one, the doc drifts and nobody can tell. + +## Claim safety + +Tightening the wording must not tighten the certainty. This rule outranks concision. + +- Carry each claim across at its original strength. +- Label assumptions as assumptions and unknowns as unknowns. +- Cite the source path, command, or evidence for each claim in durable text. +- Say a claim is unverified, or ask to verify it, rather than writing around it. +- Add nothing the source did not contain: no internal detail, no commitment, no date. + +Use labels where the distinction carries weight: `Fact:` `Assumption:` `Unknown:` + +Done when every claim in the output traces to a claim in the input at equal or weaker strength. + +## Refining text that already exists + +A rewrite keys off the prose it reads, so vocabulary changes and weak structure survives. Rebuild instead: + +1. Extract the claims, steps, and open questions as a bare list. +2. Pick the gear from that list, not from the old prose. +3. Write from the list. +4. Check the claim-safety criterion against the original. + +## Embedded use contract + +Other skills reach this skill with: + +> Before finalizing durable technical writing, apply `precise-technical-writing`: pick a gear, write from claims, hold claim strength steady, and structure for later retrieval. + +## Drift + +A gear holds for a few turns, then slips. Where a repo needs the style enforced instead of requested, gate on a prose linter such as [Vale](https://vale.sh) at pre-commit or `PostToolUse`. The skill sets the target; the gate holds it. diff --git a/eval/technical-writing/judge.py b/eval/technical-writing/judge.py new file mode 100644 index 0000000..6a8bd75 --- /dev/null +++ b/eval/technical-writing/judge.py @@ -0,0 +1,123 @@ +#!/usr/bin/env python3 +"""Blind pairwise judging for the technical-writing eval. + +The deterministic scorer measures proxies. Only a reader can judge whether an output +is actually unambiguous, so this builds pairwise comparisons a model can judge without +seeing which arm produced what. + +Blinding, because both biases are real and both would favor the skill: + - arm labels are stripped + - presentation order is shuffled with a fixed seed, so runs are reproducible + - the mapping is written to pairs/key.json, which the judge never reads + +Judged on the criteria the eval is actually about, with fidelity dominant. +""" +import argparse +import itertools +import json +import random +from pathlib import Path + +RUBRIC = """You are comparing two versions of the same technical artifact, written from +the same source notes. You do not know who or what wrote either one. + +The source notes are given first. Read them carefully: several claims in them are +explicitly uncertain, and some details are internal. + +Judge in this order. Earlier criteria dominate later ones. + +1. FIDELITY (dominant). Does the version assert anything the notes do not support? + Does it turn an uncertain claim into a confident one? Does it invent a date, + a cause, or a commitment? Does it leak an internal name into text meant to be + external? Does it flatten a disagreement into a decision? A single fidelity + failure outweighs any amount of stylistic polish. + +2. AMBIGUITY. Could a competent engineer reading this at 3am act on it and get it + wrong? Look for unclear referents, missing actors, conditions stated after the + actions they govern, and unnamed failure states. + +3. RETRIEVABILITY. If someone needs one fact from this in six months, can they find + it without reading the whole thing? + +4. REGISTER. Is it free of marketing tone, filler openers, and signposting that + carries no information? + +Explicitly NOT criteria: + - Length. Shorter is not better. Splitting one complex sentence into three simple + ones is usually an improvement even though it adds words. + - Confidence of tone. A version that says "we have not confirmed this" is better + than one that sounds authoritative, if the notes did not confirm it. + +Respond as JSON only: + +{"winner": "A" | "B" | "tie", + "fidelity_failures": {"A": ["..."], "B": ["..."]}, + "reason": "one or two sentences", + "confidence": "high" | "medium" | "low"} +""" + + +def load_task_input(here: Path, task_id: str) -> str: + raw = (here / "tasks" / f"{task_id}.md").read_text().split("---", 2) + return raw[2].strip() + + +def main() -> None: + ap = argparse.ArgumentParser() + ap.add_argument("out_dir", type=Path) + ap.add_argument("--seed", type=int, default=20260803) + ap.add_argument("--baseline-arm", default=None, + help="compare every arm against this one instead of all pairs") + args = ap.parse_args() + + here = Path(__file__).parent + rng = random.Random(args.seed) + + outputs = {} + for f in sorted(args.out_dir.glob("*__*.txt")): + task_id, arm = f.stem.split("__", 1) + outputs.setdefault(task_id, {})[arm] = f.read_text().strip() + + pairs_dir = args.out_dir / "pairs" + pairs_dir.mkdir(exist_ok=True) + for stale in pairs_dir.glob("*"): + stale.unlink() + + key = {} + n = 0 + for task_id, by_arm in sorted(outputs.items()): + arms = sorted(by_arm) + combos = ([(args.baseline_arm, a) for a in arms if a != args.baseline_arm] + if args.baseline_arm else list(itertools.combinations(arms, 2))) + for left, right in combos: + if left not in by_arm or right not in by_arm: + continue + shown = [(left, by_arm[left]), (right, by_arm[right])] + rng.shuffle(shown) + pair_id = f"{task_id}__{n:03d}" + key[pair_id] = {"task": task_id, "A": shown[0][0], "B": shown[1][0]} + (pairs_dir / f"{pair_id}.txt").write_text( + f"{RUBRIC}\n" + f"=== SOURCE NOTES ===\n\n{load_task_input(here, task_id)}\n\n" + f"=== VERSION A ===\n\n{shown[0][1]}\n\n" + f"=== VERSION B ===\n\n{shown[1][1]}\n" + ) + n += 1 + + (pairs_dir / "key.json").write_text(json.dumps(key, indent=2)) + print(f"wrote {n} blinded pairs to {pairs_dir}") + print(f"mapping in {pairs_dir / 'key.json'} — do not feed it to the judge\n") + print("judge them with:") + print(f""" + for p in {pairs_dir}/*__*.txt; do + echo "=== $(basename "$p" .txt)" + ( cd "$(mktemp -d)" && claude -p --model opus --tools "" \\ + --no-session-persistence "$(cat "$p")" ) + done | tee {args.out_dir}/verdicts.txt +""") + print(f"then resolve labels: python3 {Path(__file__).name} " + f"{args.out_dir} --resolve {args.out_dir}/verdicts.txt") + + +if __name__ == "__main__": + main() diff --git a/eval/technical-writing/run.sh b/eval/technical-writing/run.sh new file mode 100755 index 0000000..930b616 --- /dev/null +++ b/eval/technical-writing/run.sh @@ -0,0 +1,107 @@ +#!/usr/bin/env bash +# Generate one output per (task, arm) pair via headless claude. +# +# Runs from a scratch cwd so no project CLAUDE.md is discovered. The operator's +# global ~/.claude/CLAUDE.md still loads unless --bare is available; see README.md +# under "Known confound". Mode is recorded in out/manifest.json. +set -euo pipefail + +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +MODEL="sonnet" +OUT="$HERE/out" +ONLY_TASKS="" +ONLY_ARMS="" +BARE="" + +while [[ $# -gt 0 ]]; do + case "$1" in + --model) MODEL="$2"; shift 2 ;; + --tasks) ONLY_TASKS="$2"; shift 2 ;; + --arms) ONLY_ARMS="$2"; shift 2 ;; + --out) OUT="$2"; shift 2 ;; + --bare) BARE="1"; shift ;; + *) echo "unknown option: $1" >&2; exit 2 ;; + esac +done + +if [[ -n "$BARE" && -z "${ANTHROPIC_API_KEY:-}" ]]; then + echo "--bare requires ANTHROPIC_API_KEY (bare mode never reads OAuth or keychain)." >&2 + exit 2 +fi + +SCRATCH="$(mktemp -d)" +trap 'rm -rf "$SCRATCH"' EXIT +mkdir -p "$OUT" + +matches() { [[ -z "$2" ]] || [[ ",$2," == *",$1,"* ]]; } + +count=0 +for task_file in "$HERE"/tasks/*.md; do + task="$(basename "$task_file" .md)" + matches "$task" "$ONLY_TASKS" || continue + + # Split the task file: YAML frontmatter is config, everything after is the prompt. + brief="$(python3 -c " +import sys, yaml +raw = open(sys.argv[1]).read().split('---', 2) +meta = yaml.safe_load(raw[1]) +print(meta['brief'].strip()) +" "$task_file")" + body="$(python3 -c " +import sys +print(open(sys.argv[1]).read().split('---', 2)[2].strip()) +" "$task_file")" + + for arm_file in "$HERE"/arms/*.txt; do + arm="$(basename "$arm_file" .txt)" + matches "$arm" "$ONLY_ARMS" || continue + + dest="$OUT/${task}__${arm}.txt" + if [[ -s "$dest" ]]; then + echo "skip $task / $arm (exists)" + continue + fi + + prompt="$brief + +$body + +Output only the finished artifact. No preamble, no explanation of your choices." + + sys="$(cat "$arm_file")" + # The prompt goes in on stdin, never as a positional argument: --tools is + # variadic, so `--tools "" "$prompt"` silently swallows the prompt. That bug + # hit only the baseline arm, whose flag list ends with --tools. + cmd=(claude -p --model "$MODEL" --no-session-persistence --tools "") + [[ -n "$BARE" ]] && cmd+=(--bare) + # An empty system prompt would be rejected; baseline gets no flag at all. + [[ -n "$sys" ]] && cmd+=(--append-system-prompt "$sys") + + echo "gen $task / $arm" + ( cd "$SCRATCH" && printf '%s' "$prompt" | "${cmd[@]}" ) > "$dest" || { + echo "FAILED $task / $arm" >&2 + rm -f "$dest" + continue + } + count=$((count + 1)) + done +done + +python3 - "$OUT" "$MODEL" "${BARE:-0}" <<'PY' +import json, os, subprocess, sys +out, model, bare = sys.argv[1], sys.argv[2], sys.argv[3] == "1" +rev = subprocess.run(["git", "rev-parse", "--short", "HEAD"], + capture_output=True, text=True).stdout.strip() +json.dump({ + "model": model, + "bare": bare, + "clean_room": bare, + "skill_revision": rev, + "note": ("clean room" if bare else + "operator global CLAUDE.md present in all arms; relative comparisons only"), + "outputs": sorted(f for f in os.listdir(out) if f.endswith(".txt")), +}, open(os.path.join(out, "manifest.json"), "w"), indent=2) +PY + +echo "generated $count new output(s) into $OUT" +echo "next: python3 $HERE/score.py $OUT --markdown" diff --git a/eval/technical-writing/score.py b/eval/technical-writing/score.py new file mode 100755 index 0000000..8d091fc --- /dev/null +++ b/eval/technical-writing/score.py @@ -0,0 +1,257 @@ +#!/usr/bin/env python3 +"""Deterministic scoring for the technical-writing eval. + +Two metric families, deliberately separated: + + STYLE proxies for ambiguity and slop. Every instructed arm should improve these. + FIDELITY whether the arm invented, hardened, or leaked a claim. Only the skill has + rules aimed at these, so this is where H2 is decided. + +Length is reported and never scored. ASD-STE100 rule 4.2 forbids shortening by +omission, so a shorter output is not a better one. + +A FIDELITY failure is not tradeable against a STYLE win: the two are reported +separately and never summed into one number. +""" +import argparse +import json +import re +import sys +from collections import defaultdict +from pathlib import Path + +import yaml + +# Slop markers. Sourced from the HN thread's own complaints (heavy signposting, +# meta-commentary, marketing register) plus the usual AI tells. +TELLS = [ + r"\bit'?s worth noting\b", r"\bit'?s important to (note|remember)\b", + r"\bthat said\b", r"\bat the end of the day\b", r"\bin today'?s\b", + r"\bdelve into\b", r"\bnavigat(e|ing) the\b", r"\bunlock(s|ing)?\b", + r"\bseamless(ly)?\b", r"\brobust\b", r"\bcomprehensive\b", r"\bleverag(e|ing)\b", + r"\butili[sz](e|ing)\b", r"\bfacilitat(e|ing)\b", r"\borchestrat(e|ing)\b", + r"\bstreamlin(e|ing)\b", r"\bcutting[- ]edge\b", r"\bbest practices?\b", + r"\bgame[- ]chang(er|ing)\b", r"\bplays? a (key|vital|crucial|central) role\b", + r"\bcritical component\b", r"\bhere'?s (the|what|why|how)\b", + r"\bthe (key|real) (insight|takeaway|question) (is|here)\b", + r"\bnot (just|only) \w+ (but|—)\b", r"\bmoving forward\b", +] + +HEDGES = [ + r"\bmight\b", r"\bperhaps\b", r"\barguably\b", r"\bsomewhat\b", r"\bfairly\b", + r"\bquite\b", r"\brelatively\b", r"\bgenerally\b", r"\btypically\b", + r"\busually\b", r"\bin some cases\b", r"\bcould potentially\b", r"\bit seems\b", +] + +# Passive voice with no named actor: "is performed", "was updated" not followed by "by". +PASSIVE = re.compile( + r"\b(?:is|are|was|were|be|been|being)\s+(\w+(?:ed|en))\b(?!\s+by\b)", re.I) + +# Abstract nouns standing where a verb belongs. +NOMINALIZATION = re.compile( + r"\b\w{4,}(?:tion|sion|ment|ance|ence|ity|ness)\b", re.I) + +CODE_FENCE = re.compile(r"```.*?```", re.S) +INLINE_CODE = re.compile(r"`[^`]*`") + + +def prose_only(text: str) -> str: + """Strip code blocks. Style rules apply to prose, not to the code being documented.""" + return INLINE_CODE.sub(" ", CODE_FENCE.sub(" ", text)) + + +def sentences(text: str) -> list[str]: + # Skip list markers and headings; they are structure, not sentences. + lines = [ln for ln in text.splitlines() + if ln.strip() and not ln.lstrip().startswith("#")] + joined = " ".join(lines) + parts = re.split(r"(?<=[.!?])\s+(?=[A-Z(\[])", joined) + return [p.strip() for p in parts if len(p.split()) >= 3] + + +def count_patterns(text: str, patterns: list[str]) -> int: + return sum(len(re.findall(p, text, re.I)) for p in patterns) + + +def per_kw(n: int, words: int) -> float: + """Rate per 1000 words, so a longer output is not penalized for being longer.""" + return round(n * 1000 / words, 1) if words else 0.0 + + +def load_task(path: Path) -> dict: + raw = path.read_text().split("---", 2) + meta = yaml.safe_load(raw[1]) + meta["input"] = raw[2] + return meta + + +def score_style(text: str) -> dict: + prose = prose_only(text) + words = len(prose.split()) + sents = sentences(prose) + lengths = sorted(len(s.split()) for s in sents) + + def pct(p): + return lengths[min(int(len(lengths) * p), len(lengths) - 1)] if lengths else 0 + + return { + "words": words, + "sentences": len(sents), + "median_sentence_words": pct(0.5), + "p90_sentence_words": pct(0.9), + "over_20_words_pct": round( + 100 * sum(1 for n in lengths if n > 20) / len(lengths), 1) if lengths else 0.0, + "tells_per_1k": per_kw(count_patterns(prose, TELLS), words), + "hedges_per_1k": per_kw(count_patterns(prose, HEDGES), words), + "agentless_passive_per_1k": per_kw(len(PASSIVE.findall(prose)), words), + "nominalizations_per_1k": per_kw(len(NOMINALIZATION.findall(prose)), words), + } + + +def score_fidelity(text: str, task: dict) -> dict: + """Did the arm invent, harden, leak, or flatten a claim? + + Every finding names the task rule it broke, so a failure is auditable rather + than a number to trust. + """ + findings = [] + + for rule in task.get("forbidden") or []: + hits = re.findall(rule["pattern"], text, re.M) + if hits: + findings.append({ + "kind": "forbidden", + "why": rule["why"], + "matched": list({h if isinstance(h, str) else h[0] for h in hits})[:3], + }) + + for rule in task.get("must_hedge") or []: + if not any(re.search(rf"\b{re.escape(m)}", text, re.I) for m in rule["markers"]): + findings.append({ + "kind": "claim_hardened", + "claim": rule["claim"], + "why": rule["why"], + "matched": [], + }) + + for rule in task.get("required") or []: + if not re.search(rule["pattern"], text, re.M): + findings.append({ + "kind": "omitted", + "why": rule["why"], + "matched": [], + }) + + drift = [] + for group in task.get("terms") or []: + used = [t for t in group if re.search(rf"\b{re.escape(t)}\b", text, re.I)] + if len(used) > 1: + drift.append(used) + if drift: + findings.append({ + "kind": "term_drift", + "why": "same concept named more than one way", + "matched": [" / ".join(g) for g in drift], + }) + + return { + "failures": len(findings), + "claim_hardened": sum(1 for f in findings if f["kind"] == "claim_hardened"), + "forbidden": sum(1 for f in findings if f["kind"] == "forbidden"), + "omitted": sum(1 for f in findings if f["kind"] == "omitted"), + "term_drift": sum(1 for f in findings if f["kind"] == "term_drift"), + "findings": findings, + } + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("out_dir", type=Path) + ap.add_argument("--markdown", action="store_true") + args = ap.parse_args() + + here = Path(__file__).parent + tasks = {p.stem: load_task(p) for p in sorted((here / "tasks").glob("*.md"))} + + results = {} + for f in sorted(args.out_dir.glob("*__*.txt")): + task_id, arm = f.stem.split("__", 1) + if task_id not in tasks: + print(f"warn: no task definition for {task_id}, skipping", file=sys.stderr) + continue + text = f.read_text() + results[f.stem] = { + "task": task_id, "arm": arm, + "style": score_style(text), + "fidelity": score_fidelity(text, tasks[task_id]), + } + + if not results: + print("no outputs found — run ./run.sh first", file=sys.stderr) + return 1 + + dest = args.out_dir / "scores.json" + dest.write_text(json.dumps(results, indent=2)) + + if args.markdown: + emit_markdown(results) + print(f"\nwrote {dest}", file=sys.stderr) + return 0 + + +def emit_markdown(results: dict) -> None: + arms = sorted({r["arm"] for r in results.values()}) + tasks = sorted({r["task"] for r in results.values()}) + + print("## Fidelity failures (lower is better; this decides H2)\n") + print("| Task | " + " | ".join(arms) + " |") + print("|---" * (len(arms) + 1) + "|") + for t in tasks: + row = [] + for a in arms: + r = results.get(f"{t}__{a}") + row.append("—" if not r else str(r["fidelity"]["failures"])) + print(f"| `{t}` | " + " | ".join(row) + " |") + totals = [] + for a in arms: + totals.append(str(sum(r["fidelity"]["failures"] + for r in results.values() if r["arm"] == a))) + print("| **total** | " + " | ".join(f"**{x}**" for x in totals) + " |") + + print("\n### Failures by kind\n") + print("| Arm | claim hardened | forbidden | omitted | term drift |") + print("|---|---|---|---|---|") + for a in arms: + rs = [r for r in results.values() if r["arm"] == a] + print(f"| `{a}` | " + + " | ".join(str(sum(r["fidelity"][k] for r in rs)) + for k in ("claim_hardened", "forbidden", "omitted", "term_drift")) + + " |") + + print("\n## Style (rates per 1000 words; lower is better)\n") + keys = ["tells_per_1k", "hedges_per_1k", "agentless_passive_per_1k", + "nominalizations_per_1k", "over_20_words_pct", "median_sentence_words"] + print("| Arm | " + " | ".join(k.replace("_per_1k", "").replace("_", " ") + for k in keys) + " | words |") + print("|---" * (len(keys) + 2) + "|") + for a in arms: + rs = [r for r in results.values() if r["arm"] == a] + cells = [f"{sum(r['style'][k] for r in rs) / len(rs):.1f}" for k in keys] + wc = sum(r["style"]["words"] for r in rs) + print(f"| `{a}` | " + " | ".join(cells) + f" | {wc} |") + print("\n_Word counts are context, not score._") + + print("\n## Every finding\n") + for name in sorted(results): + fs = results[name]["fidelity"]["findings"] + if not fs: + continue + print(f"**`{name}`**\n") + for f in fs: + matched = f" — matched: `{'`, `'.join(f['matched'])}`" if f["matched"] else "" + print(f"- `{f['kind']}`: {f['why']}{matched}") + print() + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/eval/technical-writing/tasks/code-comment.md b/eval/technical-writing/tasks/code-comment.md new file mode 100644 index 0000000..248ecec --- /dev/null +++ b/eval/technical-writing/tasks/code-comment.md @@ -0,0 +1,42 @@ +--- +id: code-comment +gear: 3 +brief: > + Write the doc comment for this function. The reader is an engineer who has to call it + correctly and has not read its body. +terms: + - ["cache", "local cache", "cached set"] + - ["entry", "record", "item"] +forbidden: + - pattern: "(?i)\\b(utili[sz]e|leverage|facilitate|orchestrat)" + why: abstract verb where a concrete one exists + - pattern: "(?i)\\b(is|are|be) (performed|executed|carried out|utili[sz]ed)\\b" + why: passive voice with no actor + - pattern: "(?i)\\bthis (function|method) (is responsible for|serves to|aims to)\\b" + why: filler opener +must_hedge: [] +required: + - pattern: "(?i)(raise|throw|error|exception|ValueError)" + why: the function has a failure mode the caller must handle; gear 3 requires naming it + - pattern: "(?i)(mutat|in place|modifi|changes the|removes from|deletes from)" + why: the function mutates its argument, which is the one thing a caller must know +--- + +## Input + +```python +def reconcile(cache, redis, *, ttl=None, dry_run=False): + if ttl is not None and ttl <= 0: + raise ValueError("ttl must be positive") + live = set(redis.scan_iter(match="entry:*")) + stale = [k for k in cache if f"entry:{k}" not in live] + if dry_run: + return stale + for k in stale: + del cache[k] + for k in live - {f"entry:{k}" for k in cache}: + cache[k.removeprefix("entry:")] = redis.get(k) + if ttl: + redis.expire(k, ttl) + return stale +``` diff --git a/eval/technical-writing/tasks/incident-reply.md b/eval/technical-writing/tasks/incident-reply.md new file mode 100644 index 0000000..1cc2c59 --- /dev/null +++ b/eval/technical-writing/tasks/incident-reply.md @@ -0,0 +1,53 @@ +--- +id: incident-reply +gear: 2 +brief: > + Write the customer-facing status update for this incident, for posting on our public + status page. Use only what the internal notes support. +terms: + - ["upload", "ingest", "import"] + - ["customer", "client", "user"] +forbidden: + - pattern: "(?i)\\bkafka-ingest-3\\b" + why: internal hostname, not for a public status page + - pattern: "(?i)\\bLegacyBlobShim\\b" + why: internal class name, not for a public status page + - pattern: "(?i)root cause (is|was)\\b" + why: states an unconfirmed cause as established fact + - pattern: "(?i)(this week|by (monday|tuesday|wednesday|thursday|friday)|within \\d+ (hours|days)|by end of)" + why: invents a delivery commitment the notes do not contain + - pattern: "(?i)(no data (was )?lost|nothing was lost|all data (is|was) (safe|intact))" + why: notes say the 41 records are still unaccounted for + - pattern: "(?i)(we (deeply )?apologi[sz]e for the inconvenience|we sincerely apologi[sz]e)" + why: boilerplate the notes do not ask for and gear 2 cuts as filler +must_hedge: + - claim: cause + markers: ["likely", "appears", "suspect", "not confirmed", "unconfirmed", "possible", "may", "believe", "preliminary", "still investigating", "under investigation"] + why: the notes mark the cause as unconfirmed + - claim: missing_records + markers: ["unaccounted", "still", "not yet", "investigating", "unresolved", "outstanding"] + why: the 41 records are unresolved, not recovered +--- + +## Input + +Internal incident notes, verbatim from the on-call channel: + +``` +2026-07-29 14:02 — alerts firing on upload failures, ~8% of POST /v2/upload returning 500 +2026-07-29 14:20 — kafka-ingest-3 is the only broker showing the errors. Restarted it. + error rate drops to ~1% but doesn't clear. +2026-07-29 15:10 — sam thinks it's the LegacyBlobShim retry path double-acking and + dropping the offset. NOT CONFIRMED. we don't have the trace data to + prove it, the sampling was at 1% during the window. +2026-07-29 15:45 — rolled back to build 4471. error rate at 0% since 15:38. +2026-07-29 16:30 — reconciliation finds 41 upload records that we can't match to a + stored blob. could be the same bug, could be a reconciliation + artifact. still digging. do NOT tell anyone these are lost yet. +2026-07-30 09:15 — priya: we should fix the shim retry path but nobody has scoped it. + not on this sprint. no date. +2026-07-30 09:40 — 41 records still unaccounted for. reconciliation job rerun didn't + change the number. +``` + +Customers affected: uploads failing intermittently for roughly 100 minutes. diff --git a/eval/technical-writing/tasks/pr-description.md b/eval/technical-writing/tasks/pr-description.md new file mode 100644 index 0000000..4891d62 --- /dev/null +++ b/eval/technical-writing/tasks/pr-description.md @@ -0,0 +1,41 @@ +--- +id: pr-description +gear: 2 +brief: > + Write the pull request description for this change. +terms: + - ["connection pool", "pool"] + - ["timeout", "deadline"] +forbidden: + - pattern: "(?i)\\b(significantly|dramatically|greatly|massively) (improv|reduc|enhanc|boost)" + why: unquantified intensifier where a number exists in the input + - pattern: "(?i)\\b(robust|seamless|comprehensive|elegant|clean) (solution|implementation|approach|fix)\\b" + why: marketing register + - pattern: "(?i)this (PR|change) (introduces|brings) (a|an) (new|improved)\\b" + why: filler opener +must_hedge: + - claim: unverified_gain + markers: ["local", "not measured", "unverified", "staging", "expect", "should", "have not", "no production"] + why: the 40% figure came from a local benchmark and has not been measured in production +required: + - pattern: "(?i)(migration|deploy|order|before|first|drain)" + why: the change has a deploy-ordering hazard that a reviewer must be told about +--- + +## Input + +What changed, from the author's own notes: + +``` +- swapped the per-request psycopg connection for a shared pool (pool size 20) +- added statement_timeout=5s, previously unbounded +- deleted the retry wrapper in db/legacy.py, the pool handles it now +- local benchmark: p99 on /reports went 1.9s -> 1.1s (about 40% better). + I have NOT measured this in staging or prod. Just my laptop, 200 iterations. +- CAREFUL: the pool reads DB_POOL_SIZE from env. If you deploy this before the + config change lands in the terraform repo, it defaults to 5 and reports will + queue. The terraform PR is separate and has to go first. +- statement_timeout will now kill two known-slow admin queries that used to + finish in ~8s. Those will start erroring. I think that's correct behavior but + someone from the admin team should confirm. +``` diff --git a/eval/technical-writing/tasks/reference-summary.md b/eval/technical-writing/tasks/reference-summary.md new file mode 100644 index 0000000..15263e8 --- /dev/null +++ b/eval/technical-writing/tasks/reference-summary.md @@ -0,0 +1,52 @@ +--- +id: reference-summary +gear: 4 +brief: > + Write the reference entry for this service in our architecture index. Another agent + will read it to answer questions without reading the code. +terms: + - ["service", "API"] +forbidden: + - pattern: "(?i)\\b(plays a (key|vital|central) role|is a critical component|serves as the backbone)\\b" + why: prose padding in a lookup artifact +must_hedge: + - claim: ownership_unknown + markers: ["unknown", "unclear", "not documented", "no owner", "unowned", "TBD", "unassigned"] + why: the input says ownership is disputed and undocumented +required: + - pattern: "(?i)^#*\\s*sources?\\b" + why: gear 4 requires a Sources field so the entry can be re-checked against the code + - pattern: "(?i)(invariant|must always|guarantee)" + why: the input contains an invariant, which is the highest-value field for an agent +--- + +## Input + +Whatever anyone could remember about the service, collected in a thread: + +``` +NotificationDispatcher. Lives in services/notify/. Entry point is +Dispatcher.RunAsync in Dispatcher.cs. + +It reads off the notifications topic and fans out to email (SendGrid), push (FCM), +and SMS (Twilio). Config for which channels are on per-tenant is in +NotifyOptions.cs, loaded from app config. + +Depends on: IdentityService (to resolve a user id to contact details), TenantConfig +(channel toggles), and the three vendor SDKs. + +Important thing nobody wrote down: it must never send the same notification twice +for the same (notification_id, channel) pair. There's a dedupe table, +notify_sent_log, and the whole design assumes that constraint holds. If you add a +channel you have to add it to the dedupe key or you get duplicate sends. + +Failure modes: SendGrid 429s a lot, there's a backoff. FCM token expiry produces a +permanent failure that gets logged and dropped, deliberately. Twilio failures retry +3x then dead-letter to notify_dlq. + +Who owns it: honestly unclear. It was the Platform team, then it moved to Growth +during the reorg, but Growth says they never accepted it. Nobody has updated the +service catalog. + +Open question: nobody knows if the dedupe table is ever pruned. It's 400M rows. +``` diff --git a/eval/technical-writing/tasks/runbook.md b/eval/technical-writing/tasks/runbook.md new file mode 100644 index 0000000..6954e8f --- /dev/null +++ b/eval/technical-writing/tasks/runbook.md @@ -0,0 +1,44 @@ +--- +id: runbook +gear: 3 +brief: > + Turn these notes into the runbook step for rotating the signing key. The reader is + on-call at 3am and has not done this before. +terms: + - ["signing key", "key"] + - ["revoke", "invalidate"] +forbidden: + - pattern: "(?i)^\\s*\\d+\\..*\\bif (the|there|you)\\b.*," + why: condition placed after the action instead of before it + - pattern: "(?i)\\b(simply|just|merely) (run|execute|click|do)\\b" + why: minimizes a step that has a destructive failure mode +must_hedge: [] +required: + - pattern: "(?i)(do not|must not|never|before)" + why: there is an ordering constraint whose violation logs out every user + - pattern: "(?i)(rollback|roll back|revert|restore)" + why: the notes contain a recovery path and a 3am reader needs it +--- + +## Input + +Notes from the engineer who did it last time: + +``` +you get the new key from vault, path is secret/auth/signing, field is next_key. +it's already generated, the cron makes it monthly. + +then you set it as the active key via the admin API, PUT /admin/keys/active with +the kid. THE OLD KEY HAS TO STAY IN THE VERIFY SET or every live session breaks — +there's a separate list, verify_kids, and the old kid must be in it for at least +24h because that's the token TTL. if you revoke the old kid immediately you log +out every user, which is what happened in April. + +after 24h you remove the old kid from verify_kids. + +if the PUT fails halfway you can end up with active_kid set but verify_kids not +updated. symptom is 401s on everything. fix is PUT the old kid back as active, +then retry. + +the health endpoint /admin/keys/health shows both lists, check it after every step. +``` diff --git a/eval/technical-writing/tasks/tradeoff.md b/eval/technical-writing/tasks/tradeoff.md new file mode 100644 index 0000000..46e05b7 --- /dev/null +++ b/eval/technical-writing/tasks/tradeoff.md @@ -0,0 +1,57 @@ +--- +id: tradeoff +gear: 1 +brief: > + Write up this design discussion for the team so someone joining next week understands + where the thinking currently stands. +terms: + - ["gateway", "proxy", "edge"] +forbidden: + - pattern: "(?i)^#+ *(decision|we (will|have) (decided|chosen))" + why: flattens a live disagreement into a decision that was never made + - pattern: "(?i)\\bwe (will|have) (decided|chosen|selected|agreed)\\b" + why: no decision was reached in the input + - pattern: "(?i)\\b(the )?recommended (approach|option|path) is\\b" + why: invents a recommendation neither engineer made +must_hedge: + - claim: unresolved + markers: ["unresolved", "disagree", "open", "not decided", "undecided", "no decision", "still", "yet to", "tension", "argument"] + why: the discussion ended without agreement and that is the main fact + - claim: cost_unknown + markers: ["unknown", "unclear", "no numbers", "not measured", "guess", "estimate", "nobody", "not been"] + why: the latency cost is explicitly unmeasured +required: + - pattern: "(?i)\\b(mira|dev)\\b" + why: attributing positions to the people holding them is what makes exploration readable +--- + +## Input + +Notes from a whiteboard session. Nothing was decided. + +``` +Question: do we put the new billing API behind the existing APIM gateway, or give it +its own ingress? + +Mira's position: APIM. We already pay for it, it already does the auth handoff, and +every other service is behind it. Standing up a second ingress means a second set of +WAF rules, a second cert rotation, a second thing that breaks at 3am. She's been burned +by exactly this at a previous job — two ingresses drifted apart over a year and nobody +noticed until an audit. + +Dev's position: own ingress. APIM adds a hop we can't tune, and billing is the one +service where p99 actually shows up in a contract. He also points out APIM's policy +language is a pain to test and the billing team can't deploy a policy change without +going through the platform team, which is a two-week queue right now. + +Where they agree: the two-week platform queue is the real problem and neither option +fixes it. + +Where they got stuck: nobody has measured what the APIM hop actually costs. Dev thinks +it's 15-40ms. Mira thinks it's under 10ms. There is no number. Somebody could measure +it in an afternoon and nobody has. + +Also raised and dropped: Kong (nobody wants to operate it), and putting billing behind +APIM but with a bypass route for the one latency-sensitive endpoint. That last one got +a "huh, maybe" from both of them and then the meeting ended. +``` From b8de384f3bd04b85091da8ebda5150c14e197669 Mon Sep 17 00:00:00 2001 From: Scott Pfister Date: Mon, 3 Aug 2026 13:41:47 -0500 Subject: [PATCH 06/11] fix(eval): four scoring bugs that made the eval measure itself Found by reading the outputs behind findings that fired identically across all four arms. A rule that flags every arm is measuring the rule. - required patterns could not match through markdown emphasis, so a real "**Sources:**" field read as omitted on every arm that used bold. - must_hedge fired whenever hedge markers were absent, but they are also absent when the claim itself is absent. Hardening and omission are opposite outcomes; each must_hedge rule now declares presence markers and they are counted separately. - term groups holding a substring pair ("pool" inside "connection pool", "key" inside "signing key") flagged drift on every output by construction. Replaced, and validate_task now rejects the whole class. - score.py accepted half-written files from an in-flight generation and reported every hedge as missing. It now skips anything under 20 words. Also: the incident-reply hedge rules take absent_ok, because for a public status page saying nothing about an unconfirmed cause is restraint, not failure. The measured failure is raising the claim and stating it firmly. --- eval/technical-writing/judge.py | 73 ++++++++++++++++++- eval/technical-writing/score.py | 73 +++++++++++++++++-- eval/technical-writing/tasks/code-comment.md | 2 +- .../technical-writing/tasks/incident-reply.md | 18 ++++- .../technical-writing/tasks/pr-description.md | 5 +- .../tasks/reference-summary.md | 6 +- eval/technical-writing/tasks/runbook.md | 2 +- eval/technical-writing/tasks/tradeoff.md | 8 +- 8 files changed, 168 insertions(+), 19 deletions(-) diff --git a/eval/technical-writing/judge.py b/eval/technical-writing/judge.py index 6a8bd75..aec39a3 100644 --- a/eval/technical-writing/judge.py +++ b/eval/technical-writing/judge.py @@ -15,7 +15,10 @@ import argparse import itertools import json +import re +import sys import random +from collections import defaultdict from pathlib import Path RUBRIC = """You are comparing two versions of the same technical artifact, written from @@ -68,11 +71,17 @@ def main() -> None: ap.add_argument("--seed", type=int, default=20260803) ap.add_argument("--baseline-arm", default=None, help="compare every arm against this one instead of all pairs") + ap.add_argument("--resolve", type=Path, default=None, + help="de-blind a verdicts file produced by the judging loop") args = ap.parse_args() here = Path(__file__).parent rng = random.Random(args.seed) + if args.resolve: + resolve(args.out_dir, args.resolve) + return + outputs = {} for f in sorted(args.out_dir.glob("*__*.txt")): task_id, arm = f.stem.split("__", 1) @@ -107,17 +116,77 @@ def main() -> None: (pairs_dir / "key.json").write_text(json.dumps(key, indent=2)) print(f"wrote {n} blinded pairs to {pairs_dir}") print(f"mapping in {pairs_dir / 'key.json'} — do not feed it to the judge\n") - print("judge them with:") + print("judge them with (prompt on stdin — --tools is variadic and would eat it):") print(f""" for p in {pairs_dir}/*__*.txt; do echo "=== $(basename "$p" .txt)" ( cd "$(mktemp -d)" && claude -p --model opus --tools "" \\ - --no-session-persistence "$(cat "$p")" ) + --no-session-persistence < "$p" ) done | tee {args.out_dir}/verdicts.txt """) print(f"then resolve labels: python3 {Path(__file__).name} " f"{args.out_dir} --resolve {args.out_dir}/verdicts.txt") +def resolve(out_dir: Path, verdicts_file: Path) -> None: + """Map blinded A/B verdicts back to arm names and tally wins.""" + key = json.loads((out_dir / "pairs" / "key.json").read_text()) + text = verdicts_file.read_text() + + wins = defaultdict(int) + losses = defaultdict(int) + ties = defaultdict(int) + rows = [] + + # Blocks are delimited by the "=== " lines the judging loop echoes. + blocks = re.split(r"^===\s*(\S+)\s*$", text, flags=re.M)[1:] + for pair_id, block in zip(blocks[::2], blocks[1::2]): + if pair_id not in key: + print(f"warn: verdict for unknown pair {pair_id}", file=sys.stderr) + continue + m = re.search(r"\{.*\}", block, re.S) + if not m: + print(f"warn: no JSON verdict in block {pair_id}", file=sys.stderr) + continue + try: + v = json.loads(m.group(0)) + except json.JSONDecodeError: + print(f"warn: unparseable verdict in block {pair_id}", file=sys.stderr) + continue + + mapping = key[pair_id] + winner = v.get("winner") + if winner == "tie": + ties[mapping["A"]] += 1 + ties[mapping["B"]] += 1 + won, lost = None, None + elif winner in ("A", "B"): + won = mapping[winner] + lost = mapping["B" if winner == "A" else "A"] + wins[won] += 1 + losses[lost] += 1 + else: + print(f"warn: verdict {winner!r} in {pair_id} is not A/B/tie", file=sys.stderr) + continue + + rows.append({ + "pair": pair_id, "task": mapping["task"], + "winner": won, "loser": lost, + "confidence": v.get("confidence"), "reason": v.get("reason"), + "fidelity_failures": {mapping[k]: v.get("fidelity_failures", {}).get(k, []) + for k in ("A", "B")}, + }) + + dest = out_dir / "verdicts.json" + dest.write_text(json.dumps(rows, indent=2)) + + arms = sorted(set(list(wins) + list(losses) + list(ties))) + print("| Arm | wins | losses | ties |") + print("|---|---|---|---|") + for a in arms: + print(f"| `{a}` | {wins[a]} | {losses[a]} | {ties[a]} |") + print(f"\njudged {len(rows)} pairs; detail in {dest}") + + if __name__ == "__main__": main() diff --git a/eval/technical-writing/score.py b/eval/technical-writing/score.py index 8d091fc..21091de 100755 --- a/eval/technical-writing/score.py +++ b/eval/technical-writing/score.py @@ -54,6 +54,20 @@ CODE_FENCE = re.compile(r"```.*?```", re.S) INLINE_CODE = re.compile(r"`[^`]*`") +# Markdown emphasis around a field name hid it from the `required` patterns: +# "**Sources:**" never matched /^#*\s*sources/. Normalize before matching. +EMPHASIS = re.compile(r"[*_]{1,3}") + + +def normalize(text: str) -> str: + """Strip markdown emphasis and list markers so field-name patterns can anchor.""" + lines = [] + for ln in text.splitlines(): + ln = EMPHASIS.sub("", ln) + ln = re.sub(r"^\s*[-*+]\s+", "", ln) + lines.append(ln.strip()) + return "\n".join(lines) + def prose_only(text: str) -> str: """Strip code blocks. Style rules apply to prose, not to the code being documented.""" @@ -82,9 +96,32 @@ def load_task(path: Path) -> dict: raw = path.read_text().split("---", 2) meta = yaml.safe_load(raw[1]) meta["input"] = raw[2] + validate_task(path.stem, meta) return meta +def validate_task(task_id: str, meta: dict) -> None: + """Reject rules that fire on every output regardless of arm. + + A term group holding both "pool" and "connection pool" always reports drift, + because matching the long form also matches the short one. Such a rule measures + the rule, not the arm. + """ + for group in meta.get("terms") or []: + low = [t.lower() for t in group] + for a in low: + for b in low: + if a != b and a in b: + raise SystemExit( + f"{task_id}: term group {group} has '{a}' inside '{b}', so it " + f"flags every output. Use terms that are not substrings.") + for rule in meta.get("must_hedge") or []: + if "presence" not in rule: + raise SystemExit( + f"{task_id}: must_hedge rule '{rule.get('claim')}' needs a 'presence' " + f"list, else an omitted claim is miscounted as a hardened one.") + + def score_style(text: str) -> dict: prose = prose_only(text) words = len(prose.split()) @@ -115,9 +152,10 @@ def score_fidelity(text: str, task: dict) -> dict: than a number to trust. """ findings = [] + norm = normalize(text) for rule in task.get("forbidden") or []: - hits = re.findall(rule["pattern"], text, re.M) + hits = re.findall(rule["pattern"], norm, re.M) if hits: findings.append({ "kind": "forbidden", @@ -126,16 +164,30 @@ def score_fidelity(text: str, task: dict) -> dict: }) for rule in task.get("must_hedge") or []: - if not any(re.search(rf"\b{re.escape(m)}", text, re.I) for m in rule["markers"]): + # An absent claim has no hedge markers either, so absence and hardening look + # identical unless presence is tested first. They are opposite outcomes: + # hardening asserts something unsupported, omission just leaves it out. + present = any(re.search(rf"\b{re.escape(m)}", norm, re.I) + for m in rule["presence"]) + hedged = any(re.search(rf"\b{re.escape(m)}", norm, re.I) + for m in rule["markers"]) + if present and not hedged: findings.append({ "kind": "claim_hardened", "claim": rule["claim"], "why": rule["why"], "matched": [], }) + elif not present and not rule.get("absent_ok", False): + findings.append({ + "kind": "claim_absent", + "claim": rule["claim"], + "why": f"{rule['why']} (claim not raised at all)", + "matched": [], + }) for rule in task.get("required") or []: - if not re.search(rule["pattern"], text, re.M): + if not re.search(rule["pattern"], norm, re.M): findings.append({ "kind": "omitted", "why": rule["why"], @@ -157,6 +209,7 @@ def score_fidelity(text: str, task: dict) -> dict: return { "failures": len(findings), "claim_hardened": sum(1 for f in findings if f["kind"] == "claim_hardened"), + "claim_absent": sum(1 for f in findings if f["kind"] == "claim_absent"), "forbidden": sum(1 for f in findings if f["kind"] == "forbidden"), "omitted": sum(1 for f in findings if f["kind"] == "omitted"), "term_drift": sum(1 for f in findings if f["kind"] == "term_drift"), @@ -180,6 +233,13 @@ def main() -> int: print(f"warn: no task definition for {task_id}, skipping", file=sys.stderr) continue text = f.read_text() + # An in-flight generation truncates its destination before filling it, so a + # concurrent score run sees an empty file and reports every hedge as missing. + # Refuse rather than emit a plausible wrong number. + if len(text.split()) < 20: + print(f"warn: {f.name} has {len(text.split())} words — partial or failed " + f"generation, skipping", file=sys.stderr) + continue results[f.stem] = { "task": task_id, "arm": arm, "style": score_style(text), @@ -219,13 +279,14 @@ def emit_markdown(results: dict) -> None: print("| **total** | " + " | ".join(f"**{x}**" for x in totals) + " |") print("\n### Failures by kind\n") - print("| Arm | claim hardened | forbidden | omitted | term drift |") - print("|---|---|---|---|---|") + print("| Arm | claim hardened | claim absent | forbidden | omitted | term drift |") + print("|---|---|---|---|---|---|") for a in arms: rs = [r for r in results.values() if r["arm"] == a] print(f"| `{a}` | " + " | ".join(str(sum(r["fidelity"][k] for r in rs)) - for k in ("claim_hardened", "forbidden", "omitted", "term_drift")) + for k in ("claim_hardened", "claim_absent", "forbidden", + "omitted", "term_drift")) + " |") print("\n## Style (rates per 1000 words; lower is better)\n") diff --git a/eval/technical-writing/tasks/code-comment.md b/eval/technical-writing/tasks/code-comment.md index 248ecec..46062dc 100644 --- a/eval/technical-writing/tasks/code-comment.md +++ b/eval/technical-writing/tasks/code-comment.md @@ -5,7 +5,7 @@ brief: > Write the doc comment for this function. The reader is an engineer who has to call it correctly and has not read its body. terms: - - ["cache", "local cache", "cached set"] + - ["stale entries", "orphaned entries", "dangling entries"] - ["entry", "record", "item"] forbidden: - pattern: "(?i)\\b(utili[sz]e|leverage|facilitate|orchestrat)" diff --git a/eval/technical-writing/tasks/incident-reply.md b/eval/technical-writing/tasks/incident-reply.md index 1cc2c59..c512ade 100644 --- a/eval/technical-writing/tasks/incident-reply.md +++ b/eval/technical-writing/tasks/incident-reply.md @@ -5,7 +5,9 @@ brief: > Write the customer-facing status update for this incident, for posting on our public status page. Use only what the internal notes support. terms: - - ["upload", "ingest", "import"] + # "ingest" is excluded deliberately: it appears inside the forbidden internal + # hostname, so counting it here would double-charge the same mistake. + - ["outage", "incident", "disruption"] - ["customer", "client", "user"] forbidden: - pattern: "(?i)\\bkafka-ingest-3\\b" @@ -21,11 +23,21 @@ forbidden: - pattern: "(?i)(we (deeply )?apologi[sz]e for the inconvenience|we sincerely apologi[sz]e)" why: boilerplate the notes do not ask for and gear 2 cuts as filler must_hedge: + # absent_ok on both: for a public status page, saying nothing about an unconfirmed + # cause or about records the notes forbid calling lost is defensible restraint. + # The failure being measured is raising the claim and stating it confidently. - claim: cause - markers: ["likely", "appears", "suspect", "not confirmed", "unconfirmed", "possible", "may", "believe", "preliminary", "still investigating", "under investigation"] + presence: ["cause", "reason", "shim", "retry", "offset", "trigger"] + markers: ["likely", "appears", "suspect", "not confirmed", "unconfirmed", "possible", + "may", "believe", "preliminary", "investigat", "determin", "identif", + "not yet", "continue", "continuing"] + absent_ok: true why: the notes mark the cause as unconfirmed - claim: missing_records - markers: ["unaccounted", "still", "not yet", "investigating", "unresolved", "outstanding"] + presence: ["41", "unaccounted", "data loss", "lost", "missing"] + markers: ["unaccounted", "still", "not yet", "investigat", "unresolved", + "outstanding", "reconcil", "review"] + absent_ok: true why: the 41 records are unresolved, not recovered --- diff --git a/eval/technical-writing/tasks/pr-description.md b/eval/technical-writing/tasks/pr-description.md index 4891d62..7031fa7 100644 --- a/eval/technical-writing/tasks/pr-description.md +++ b/eval/technical-writing/tasks/pr-description.md @@ -4,7 +4,6 @@ gear: 2 brief: > Write the pull request description for this change. terms: - - ["connection pool", "pool"] - ["timeout", "deadline"] forbidden: - pattern: "(?i)\\b(significantly|dramatically|greatly|massively) (improv|reduc|enhanc|boost)" @@ -15,7 +14,9 @@ forbidden: why: filler opener must_hedge: - claim: unverified_gain - markers: ["local", "not measured", "unverified", "staging", "expect", "should", "have not", "no production"] + presence: ["40", "1.1", "p99", "faster", "improvement", "benchmark", "latency"] + markers: ["local", "not measured", "unverified", "staging", "expect", "should", + "have not", "no production", "laptop", "only", "unconfirmed"] why: the 40% figure came from a local benchmark and has not been measured in production required: - pattern: "(?i)(migration|deploy|order|before|first|drain)" diff --git a/eval/technical-writing/tasks/reference-summary.md b/eval/technical-writing/tasks/reference-summary.md index 15263e8..bdec853 100644 --- a/eval/technical-writing/tasks/reference-summary.md +++ b/eval/technical-writing/tasks/reference-summary.md @@ -5,13 +5,15 @@ brief: > Write the reference entry for this service in our architecture index. Another agent will read it to answer questions without reading the code. terms: - - ["service", "API"] + - ["dedupe table", "dedupe log", "sent log"] forbidden: - pattern: "(?i)\\b(plays a (key|vital|central) role|is a critical component|serves as the backbone)\\b" why: prose padding in a lookup artifact must_hedge: - claim: ownership_unknown - markers: ["unknown", "unclear", "not documented", "no owner", "unowned", "TBD", "unassigned"] + presence: ["owner", "Platform", "Growth", "catalog", "team"] + markers: ["unknown", "unclear", "not documented", "no owner", "unowned", "TBD", + "unassigned", "disputed", "contested", "never accepted"] why: the input says ownership is disputed and undocumented required: - pattern: "(?i)^#*\\s*sources?\\b" diff --git a/eval/technical-writing/tasks/runbook.md b/eval/technical-writing/tasks/runbook.md index 6954e8f..14141ca 100644 --- a/eval/technical-writing/tasks/runbook.md +++ b/eval/technical-writing/tasks/runbook.md @@ -5,7 +5,7 @@ brief: > Turn these notes into the runbook step for rotating the signing key. The reader is on-call at 3am and has not done this before. terms: - - ["signing key", "key"] + - ["verify set", "verify_kids", "verification list"] - ["revoke", "invalidate"] forbidden: - pattern: "(?i)^\\s*\\d+\\..*\\bif (the|there|you)\\b.*," diff --git a/eval/technical-writing/tasks/tradeoff.md b/eval/technical-writing/tasks/tradeoff.md index 46e05b7..d983700 100644 --- a/eval/technical-writing/tasks/tradeoff.md +++ b/eval/technical-writing/tasks/tradeoff.md @@ -15,10 +15,14 @@ forbidden: why: invents a recommendation neither engineer made must_hedge: - claim: unresolved - markers: ["unresolved", "disagree", "open", "not decided", "undecided", "no decision", "still", "yet to", "tension", "argument"] + presence: ["APIM", "ingress", "option", "position", "approach"] + markers: ["unresolved", "disagree", "open", "not decided", "undecided", + "no decision", "still", "yet to", "tension", "stuck", "nothing was decided"] why: the discussion ended without agreement and that is the main fact - claim: cost_unknown - markers: ["unknown", "unclear", "no numbers", "not measured", "guess", "estimate", "nobody", "not been"] + presence: ["latency", "hop", "ms", "p99", "measur", "cost"] + markers: ["unknown", "unclear", "no numbers", "not measured", "unmeasured", "guess", + "estimate", "nobody", "not been", "no one", "no data"] why: the latency cost is explicitly unmeasured required: - pattern: "(?i)\\b(mira|dev)\\b" From 3555d063f2688faa67139707a556c8a72af9320d Mon Sep 17 00:00:00 2001 From: Scott Pfister Date: Mon, 3 Aug 2026 13:48:54 -0500 Subject: [PATCH 07/11] fix(precise-technical-writing): stop the completeness rules driving invention MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The eval found the skill fabricating more than the arms without it. Blind judging put it 5-12 against baseline, a one-line prompt, and Orwell's rules, and 9 of the 12 losses gave the same reason. Three fabrications, each checked against the task input by hand: tradeoff "**Date:** 2026-08-03" input has no date reference-summary "exactly-once delivery" input has a dedupe table runbook "if not present, add it now" conflicts with the documented recovery path All three break the skill's own rule against adding what the source lacks, and the runbook one invents a procedure that contradicts the real recovery. Diagnosis: "name each error state", "use these fields", and "always include Sources" each ask for a slot. With no source data for the slot, filling it means inventing, and nothing said an empty slot was allowed. The arms with no completeness rules had nothing pushing them to fill anything. The skill's only deterministic win — the sole arm to emit a Sources field — came from the same pressure as its losses. So claim safety now outranks every structural rule rather than only concision, an unsupported field is written as Unknown: instead of filled, and the three rules ask the writer to look for a fact rather than supply one. The skill did win incident-reply 3-0, the task built for claim safety, where the other arms hardened the unconfirmed cause and leaked internal detail. The deterministic scorer missed that entirely; only the judge saw it. --- eval/technical-writing/FINDINGS.md | 107 ++++++++++++++++++++++ skills/precise-technical-writing/SKILL.md | 12 ++- 2 files changed, 116 insertions(+), 3 deletions(-) create mode 100644 eval/technical-writing/FINDINGS.md diff --git a/eval/technical-writing/FINDINGS.md b/eval/technical-writing/FINDINGS.md new file mode 100644 index 0000000..c40f244 --- /dev/null +++ b/eval/technical-writing/FINDINGS.md @@ -0,0 +1,107 @@ +# Findings — run 1 + +- Model: `sonnet`, 6 tasks x 4 arms = 24 generations, one per cell +- Judge: `opus`, 18 blinded pairs, every other arm against `skill` +- Skill revision: `65cf5b0` +- Clean room: no. See "Known confound" in README.md. + +## Result + +The skill wins the task it was designed for and loses everywhere else. + +| Task | Judge verdict vs `skill` | Deterministic failures (`skill` / others) | +| ------------------- | ------------------------ | ----------------------------------------- | +| `incident-reply` | **skill 3–0** | 0 / 1, 0, 0 | +| `code-comment` | skill 2–1 | 0 / 0, 0, 0 | +| `pr-description` | skill 0–2 | 0 / 0, 0, 0 | +| `reference-summary` | skill 0–3 | **0 / 1, 1, 1** | +| `runbook` | skill 0–3 | **0 / 0, 1, 1** | +| `tradeoff` | skill 0–3 | 0 / 0, 0, 0 | +| **total** | **skill 5–12** | 0 / 2, 2, 2 | + +## H1 — any instruction beats none: partly + +Sentence length improved. `orwell` took the best median at 13.2 words against +`baseline`'s 17.0, for about 120 tokens. + +Two results went the other way: + +- The skill has the field's **worst** nominalization rate (28.3 per 1000 words against + `baseline`'s 22.3) and worst agentless passive rate (5.1 against 4.1), despite gear 3 + telling it to use concrete verbs and explicit subjects. Those two rules did not take. +- On `runbook`, `oneliner` and `orwell` both **dropped the recovery path** — the "PUT + the old kid back as active, then retry" instruction. `baseline` kept it. Instructing + for brevity destroyed information that no instruction at all preserved. This is + ASD-STE100 rule 4.2 (no shortening by omission) observed rather than argued. + +## H2 — only the skill holds claim strength: yes, and only the judge can see it + +On `incident-reply` the skill won all three pairs, and the judge named the failures: + +- `baseline` "converts an explicitly unconfirmed root-cause hypothesis into a claim + that the team 'identified the issue,' and adds internal infrastructure detail" +- `orwell` "claims the issue was identified and fully resolved when the notes + explicitly mark the cause unconfirmed and leave 41 records unreconciled" +- `oneliner` "softens the rollback's causal role and exposes internal cluster detail" + +**The deterministic scorer found none of this.** It scored all four arms clean on that +task, because every arm contained a hedge word somewhere and regex cannot tell a hedge +attached to the cause from a hedge attached to anything else. Any future run should +treat the deterministic fidelity column as a smoke test and the judge as the instrument. + +## The finding that matters: the skill fabricates more than the arms without it + +The skill lost 12 pairs, and across 9 of them the judge gave the same reason. Three +fabrications, each verified against the task input by hand: + +| Output | Invented | Source says | +| ------------------------ | ------------------------------------------------- | ---------------------------------------------------------------------- | +| `tradeoff__skill` | `**Date:** 2026-08-03` | no date anywhere in the input | +| `reference-summary__skill` | "Enforce exactly-once delivery" | a dedupe table and a never-send-twice constraint, which is not a guarantee | +| `runbook__skill` | "If it is not present, add it now." | recovery is "PUT the old kid back as active, then retry" | + +Each one violates the skill's own top-priority rule — *add nothing the source did not +contain: no internal detail, no commitment, no date* — and the third invents a +procedure that **conflicts** with the documented recovery path, which is the most +dangerous class of error in a runbook. + +The fabricated date is worth noting twice: `2026-08-03` was the date in the operator's +contaminating context, not in the task. The confound is not cosmetic; it supplied +material the model then asserted. + +### Diagnosis + +The skill's completeness rules and its claim-safety rule pull in opposite directions, +and completeness was winning: + +- gear 3: "Name each error state and its consequence" +- gear 4: "Use these fields where they apply", "Always include a `Sources` field" + +Each is an instruction to produce a slot. With no source data for the slot, filling it +means inventing. The arms without those rules had nothing pushing them to fill +anything, so they invented less. The skill's one deterministic win — being the only arm +to emit a `Sources` field — comes from the same pressure that produced its losses. + +Notably the skill handled `Sources` itself honestly: *"Verbal/thread report only — no +code, config, or ticket references provided. All facts above are unverified."* The rule +works when the skill is told what to do about an empty field, and fails when it is not. + +### Applied + +Claim safety now states that it outranks every structural rule, not just concision, and +adds: leave a field empty and write `Unknown:` rather than fill it. The three +completeness rules were reworded to ask the writer to *look for* a fact rather than +supply one. Not yet re-run — the next run tests whether this closes the gap. + +## Limits + +- One generation per cell, one model, one judge. The fabrication finding rests on 9 + concurring judgments and 3 hand-verified instances; the style rates rest on n=1. +- The scorer had four bugs, all found by reading outputs behind findings that fired on + every arm. Assume more remain. A rule that flags every arm is measuring the rule. +- Hedge markers are now permissive enough that `claim_hardened` is effectively + unmeasured deterministically. That is why H2 needed the judge. +- `opus` judged with a fidelity-dominant rubric it was handed. A rubric that ranked + style first would likely reverse several verdicts. +- Nothing here measures whether the skill fires, or whether a gear survives past one + turn. diff --git a/skills/precise-technical-writing/SKILL.md b/skills/precise-technical-writing/SKILL.md index d2ae565..e4742ab 100644 --- a/skills/precise-technical-writing/SKILL.md +++ b/skills/precise-technical-writing/SKILL.md @@ -84,7 +84,7 @@ Controlled English modeled on ASD-STE100, with software vocabulary in place of t - Use the same term for the same concept every time. - Use concrete verbs in place of abstract nouns. - Use ordered lists for procedures. -- Name each error state and its consequence. +- Name each error state the source describes, and its consequence. - Keep precise software terms. A clear sentence beats rule compliance. ### 4 `reference` @@ -94,8 +94,8 @@ Built for an agent to retrieve first and a human to read second. - Use stable headings and field names. - Put facts under predictable labels, where a reader finds them without reading prose. - Give explicit names, paths, commands, owners, states, and links. -- Use these fields where they apply: Purpose, Responsibilities, Inputs, Outputs, Dependencies, Invariants, Failure Modes, Open Questions. -- Always include a `Sources` field listing the paths the facts came from. A reference doc describes something it is not connected to, so the reader needs the path to re-check it. Without one, the doc drifts and nobody can tell. +- Use these fields where the source supports them, and omit the rest: Purpose, Responsibilities, Inputs, Outputs, Dependencies, Invariants, Failure Modes, Open Questions. +- Always include a `Sources` field listing the paths the facts came from. A reference doc describes something it is not connected to, so the reader needs the path to re-check it. Without one, the doc drifts and nobody can tell. Where the source gives no paths, say that instead. ## Claim safety @@ -106,9 +106,15 @@ Tightening the wording must not tighten the certainty. This rule outranks concis - Cite the source path, command, or evidence for each claim in durable text. - Say a claim is unverified, or ask to verify it, rather than writing around it. - Add nothing the source did not contain: no internal detail, no commitment, no date. +- Leave a field, step, or heading empty when the source does not support it. Write + `Unknown:` and stop. An empty field costs a reader nothing; an invented one costs + them a wrong decision. Use labels where the distinction carries weight: `Fact:` `Assumption:` `Unknown:` +This rule also outranks every structural rule below it. A gear that asks for a field, +an error state, or a recovery path is asking you to look for one, never to supply one. + Done when every claim in the output traces to a claim in the input at equal or weaker strength. ## Refining text that already exists From a91836c05787d4db5285908e54394150e8a17b89 Mon Sep 17 00:00:00 2001 From: Scott Pfister Date: Mon, 3 Aug 2026 14:03:41 -0500 Subject: [PATCH 08/11] =?UTF-8?q?test(eval):=20run=202=20=E2=80=94=20patch?= =?UTF-8?q?ed=20skill=20vs=20the=20original,=20five=20arms?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Patched skill moves 5-12 to 12-12 under blind judging, which is both a real gain and a dead heat with every arm including the one-line prompt. Against the original 231-line version it is 3-3, so the refactor is a wash on judged quality. Two results argue against the refactor: - The original produces the tightest prose in the field (18.5% of sentences over 20 words against the refactor's 33.1%, median 11.8 against 15.0), beating Orwell too. Pruning its restated style rules as duplication also removed reinforcement. - Only one of three fabrications is fixed. Both versions still write "exactly-once delivery" for what the source describes as a dedupe table, so that one is not the completeness pressure diagnosed in run 1 — the original has no mandatory-field rule and produces it anyway. Gear 3's "keep precise software terms" is the likely cause: a canonical term asserts the guarantees it carries. Competitor outputs deliberately not regenerated, so both skill versions face an identical opponent set and the comparison is paired. --- eval/technical-writing/FINDINGS.md | 74 ++++++++ eval/technical-writing/arms/skill-v0.txt | 231 +++++++++++++++++++++++ eval/technical-writing/arms/skill.txt | 12 +- 3 files changed, 314 insertions(+), 3 deletions(-) create mode 100644 eval/technical-writing/arms/skill-v0.txt diff --git a/eval/technical-writing/FINDINGS.md b/eval/technical-writing/FINDINGS.md index c40f244..4c9f96e 100644 --- a/eval/technical-writing/FINDINGS.md +++ b/eval/technical-writing/FINDINGS.md @@ -105,3 +105,77 @@ supply one. Not yet re-run — the next run tests whether this closes the gap. style first would likely reverse several verdicts. - Nothing here measures whether the skill fires, or whether a gear survives past one turn. + +--- + +# Findings — run 2 + +- Same 6 tasks. Five arms: `skill` is now the patched revision (`3555d06`, 137 lines) and + `skill-v0` is the original as submitted (`429617e`, 231 lines). +- `baseline`, `oneliner`, and `orwell` outputs were **not** regenerated, so both skill + versions face an identical competitor set. `skill` vs `skill-v0` is therefore a paired + comparison on fixed opponents. +- Judge: `opus`, 24 blinded pairs, every arm against `skill`. + +## Result: the patch helped, and the skill is now exactly average + +| Arm | wins | losses | +| ---------- | ---- | ------ | +| `skill` | 12 | 12 | +| `baseline` | 3 | 3 | +| `oneliner` | 3 | 3 | +| `orwell` | 3 | 3 | +| `skill-v0` | 3 | 3 | + +Run 1 had `skill` at 5–12. The patch moved it to 12–12 — a real gain, and also a dead +heat with every arm including the one-line prompt. + +**`skill` vs `skill-v0` is 3–3.** The 100-line refactor is a wash on judged quality. + +## The refactor lost style adherence + +| Arm | over 20 words | median sentence | agentless passive | nominalizations | +| ---------- | ------------- | --------------- | ----------------- | --------------- | +| `skill-v0` | **18.5%** | **11.8** | **4.0** | 21.0 | +| `orwell` | 29.9% | 13.2 | 4.6 | 23.0 | +| `skill` | 33.1% | 15.0 | 6.1 | **19.3** | +| `baseline` | 39.9% | 17.0 | 4.1 | 22.3 | + +The original produces the tightest prose in the field — better than the refactor on +every column but one, and better than Orwell. Plausible cause: the original restated its +style rules across the mode lists and the anti-pattern section, and pruning that +repetition as duplication also removed reinforcement. Tokens bought adherence. + +## The fabrication fix: one of three + +| Fabrication | `skill` (patched) | `skill-v0` | +| ------------------------------------------ | ----------------- | ---------- | +| runbook: invented "add it now" recovery | fixed | absent | +| reference-summary: "exactly-once delivery" | **still present** | **present** | +| tradeoff: invented `Date: 2026-08-03` | **still present** | absent | + +Both skill versions write *"Enforce exactly-once delivery per (notification_id, channel) +pair."* The source says only that the service must never send twice and keeps a dedupe +table. So this is not the completeness pressure diagnosed in run 1 — the original has no +mandatory-field rule and produces it anyway. + +**Revised diagnosis for that one.** Gear 3 says *keep precise software terms*. The +canonical term for the property being described is "exactly-once delivery", and reaching +for it is exactly what that rule asks for. But a canonical term carries the guarantees +the term implies, and this system only attempts the property through a dedupe table. +Vocabulary precision and claim strength can pull against each other, and gear 3 currently +only pushes one way. + +Candidate rule, not yet applied: *a term that names a guarantee asserts that guarantee. +Where the source describes a mechanism rather than proves a property, use the source's +own words.* + +The invented date appears only in the patched version. With n=1 per cell there is no way +to tell a regression from sampling noise. + +## Stopping the patch loop here + +Two patch cycles against 6 tasks at n=1 is the point where tuning becomes overfitting to +this task set. Further changes should wait on a defensible sample size — multiple +generations per cell, seed variance reported, and paired CIs — rather than another round +of chasing individual verdicts. diff --git a/eval/technical-writing/arms/skill-v0.txt b/eval/technical-writing/arms/skill-v0.txt new file mode 100644 index 0000000..6def750 --- /dev/null +++ b/eval/technical-writing/arms/skill-v0.txt @@ -0,0 +1,231 @@ +--- +name: concise-technical-writing +description: Use when creating, editing, or refining durable technical communication such as documentation, code comments, PR descriptions, issue bodies, review replies, handoff notes, architecture notes, runbooks, agent instructions, or technical explanations where clarity, concision, structure, and unambiguous wording matter. Apply this skill implicitly for final wording of durable artifacts. +author: + name: Scott Pfister + email: scott.pfister@7factor.io +--- + +# Concise Technical Writing + +Use this skill as a final communication pass for durable technical writing. + +The goal is precision: engineering communication that is clear, concise, explicit, and easy for humans and agents to reuse. + +## Default Behavior + +Use this skill when the output is likely to be saved, reviewed, reused, searched, pasted, committed, or used by another agent later. + +Examples: + +- Code comments +- README sections +- Architecture docs +- ADRs +- API docs +- PR descriptions +- Review replies +- Issue descriptions +- Handoff docs +- Task plans +- Runbooks +- Agent skills +- Project instructions + +For normal conversation, use the principles lightly. Keep replies natural. Apply the full refinement pass when the output is durable or precision matters. + +## Workflow + +Before finalizing durable technical writing: + +1. Identify the artifact type. +2. Identify the dominant communication intent. +3. Select a writing mode. +4. Apply section-level modes when a section has a different intent. +5. Refine for clarity, concision, and structure. +6. Preserve claim strength. Keep assumptions labeled as assumptions. +7. Check that the final text keeps the original meaning. + +## Intent Classifier + +Classify by intent first and artifact type second. + +- `instruct`: Tell someone what to do. +- `specify`: State requirements, contracts, rules, invariants, or acceptance criteria. +- `reference`: Help future lookup. +- `explain`: Help understanding. +- `justify`: Explain rationale, tradeoffs, or risk. +- `explore`: Think through unknowns or options. +- `respond`: Answer a person, especially in review or collaboration. + +## Modes + +### `auto` + +Default mode. Classify the artifact and intent, then choose the right mode. + +Use a dominant mode for the artifact. Override by section only when the section's intent clearly differs. + +### `engineering` + +Use for concise natural technical prose. + +Good for: + +- PR descriptions +- Explainers +- Review replies +- Design summaries +- Normal documentation +- Rationale that does not need a long narrative + +Rules: + +- Prefer short paragraphs. +- Remove filler and generic praise. +- Use specific nouns and verbs. +- Keep terminology consistent. +- State assumptions and limits. +- Separate summary, details, risks, and verification when useful. +- Keep a human tone when replying to people. + +### `controlled` + +Use controlled technical English inspired by ASD-STE100. Use software terminology instead of the official STE approved word list. + +Good for: + +- Procedures +- Code comments +- API docs +- Runbooks +- Acceptance criteria +- Requirements +- Contracts +- Invariants +- Implementation notes + +Rules: + +- Use active voice. +- Use explicit subjects. +- Put one action or claim in each sentence. +- State conditions before actions. +- Use the same term for the same concept. +- Prefer concrete verbs over abstract nouns. +- Remove filler, hedging, and marketing language. +- Keep sentences short when practical. +- Use ordered lists for procedures. +- Use bullets or tables for sets of facts. +- Separate facts, assumptions, recommendations, and rationale. +- Make error states and consequences explicit. + +Use software vocabulary. Keep precise terms even when they are outside aircraft-maintenance vocabulary. Prefer clear sentences over mechanical rule compliance. + +### `reference` + +Use for dense, predictable lookup material. + +Good for: + +- Architecture maps +- Module summaries +- Repo guides +- Handoff state +- Agent-facing memory +- Source indexes + +Optimize for agent retrieval first and human readability second. + +Rules: + +- Prefer stable headings and fields. +- Use sparse prose. +- Use explicit names, paths, commands, owners, states, and links. +- Group facts under predictable labels. +- Include sources when available. +- Do not hide important facts in paragraphs. + +Useful fields: + +- Purpose +- Responsibilities +- Inputs +- Outputs +- Dependencies +- Invariants +- Failure Modes +- Sources +- Verification +- Open Questions + +### `narrative` + +Use when nuance, exploration, persuasion, or historical context matters. + +Good for: + +- Design exploration +- Tradeoff discussion +- Strategy +- RFC discussion +- ADR rationale +- Persuasive review context + +Rules: + +- Keep the prose clear and concise, but allow more connective tissue. +- Preserve uncertainty and disagreement. +- Explain why options were accepted or rejected. +- Do not flatten tradeoffs into false certainty. +- Keep facts separate from opinions and recommendations. + +## Common Artifact Mapping + +- Code comment: usually `controlled`; use `engineering` only for rationale. +- PR description: usually `engineering`; use `controlled` for testing, rollout, and reviewer instructions. +- Review reply: usually `engineering`; use `controlled` for exact commitments or steps. +- Runbook: usually `controlled`; use `engineering` for background. +- Architecture index: usually `reference`; use `engineering` for short context. +- ADR: mixed. Decision and consequences use `controlled`; rationale uses `engineering`; exploration uses `narrative`. +- Handoff doc: usually `reference`; use `controlled` for next steps and commands. +- Agent skill: usually `controlled` for procedure; use `reference` for lookup tables; use `engineering` for short context. + +## Claim Safety + +Concise writing must not overstate certainty. + +- Do not strengthen claims during refinement. +- Preserve uncertainty when the source is uncertain. +- Mark assumptions explicitly. +- Mark inferences explicitly when useful. +- Do not convert guesses into facts. +- Include source paths, commands, or evidence when the artifact is durable and evidence exists. +- If an important claim is unverified, label it as unverified or ask whether to verify it. + +Use clear labels when needed: + +- `Fact:` +- `Assumption:` +- `Inference:` +- `Recommendation:` +- `Unknown:` + +## Embedded Use Contract + +Other skills can depend on this skill with this compact instruction: + +> Before finalizing durable technical writing, apply `concise-technical-writing`: classify intent, select a mode, refine for clarity and concision, preserve claim strength, and optimize structure for later retrieval when applicable. + +## Anti-Patterns + +Avoid: + +- Applying controlled mode to brainstorming or early design exploration. +- Making human replies sound like maintenance procedures. +- Removing useful nuance from rationale. +- Replacing precise software terminology with generic words. +- Hiding assumptions to make the text shorter. +- Turning every artifact into a prose essay. +- Turning every artifact into a rigid template. +- Adding headings when a short answer is enough. diff --git a/eval/technical-writing/arms/skill.txt b/eval/technical-writing/arms/skill.txt index d2ae565..e4742ab 100644 --- a/eval/technical-writing/arms/skill.txt +++ b/eval/technical-writing/arms/skill.txt @@ -84,7 +84,7 @@ Controlled English modeled on ASD-STE100, with software vocabulary in place of t - Use the same term for the same concept every time. - Use concrete verbs in place of abstract nouns. - Use ordered lists for procedures. -- Name each error state and its consequence. +- Name each error state the source describes, and its consequence. - Keep precise software terms. A clear sentence beats rule compliance. ### 4 `reference` @@ -94,8 +94,8 @@ Built for an agent to retrieve first and a human to read second. - Use stable headings and field names. - Put facts under predictable labels, where a reader finds them without reading prose. - Give explicit names, paths, commands, owners, states, and links. -- Use these fields where they apply: Purpose, Responsibilities, Inputs, Outputs, Dependencies, Invariants, Failure Modes, Open Questions. -- Always include a `Sources` field listing the paths the facts came from. A reference doc describes something it is not connected to, so the reader needs the path to re-check it. Without one, the doc drifts and nobody can tell. +- Use these fields where the source supports them, and omit the rest: Purpose, Responsibilities, Inputs, Outputs, Dependencies, Invariants, Failure Modes, Open Questions. +- Always include a `Sources` field listing the paths the facts came from. A reference doc describes something it is not connected to, so the reader needs the path to re-check it. Without one, the doc drifts and nobody can tell. Where the source gives no paths, say that instead. ## Claim safety @@ -106,9 +106,15 @@ Tightening the wording must not tighten the certainty. This rule outranks concis - Cite the source path, command, or evidence for each claim in durable text. - Say a claim is unverified, or ask to verify it, rather than writing around it. - Add nothing the source did not contain: no internal detail, no commitment, no date. +- Leave a field, step, or heading empty when the source does not support it. Write + `Unknown:` and stop. An empty field costs a reader nothing; an invented one costs + them a wrong decision. Use labels where the distinction carries weight: `Fact:` `Assumption:` `Unknown:` +This rule also outranks every structural rule below it. A gear that asks for a field, +an error state, or a recovery path is asking you to look for one, never to supply one. + Done when every claim in the output traces to a claim in the input at equal or weaker strength. ## Refining text that already exists From c43a7e4dba6882778f249e04f7b431e5d0e6ddde Mon Sep 17 00:00:00 2001 From: Scott Pfister Date: Mon, 3 Aug 2026 14:26:25 -0500 Subject: [PATCH 09/11] remove eval from PR --- eval/technical-writing/.gitignore | 1 - eval/technical-writing/FINDINGS.md | 181 ---------- eval/technical-writing/README.md | 136 -------- eval/technical-writing/arms/baseline.txt | 0 eval/technical-writing/arms/oneliner.txt | 1 - eval/technical-writing/arms/orwell.txt | 8 - eval/technical-writing/arms/skill-v0.txt | 231 ------------- eval/technical-writing/arms/skill.txt | 137 -------- eval/technical-writing/judge.py | 192 ----------- eval/technical-writing/run.sh | 107 ------ eval/technical-writing/score.py | 318 ------------------ eval/technical-writing/tasks/code-comment.md | 42 --- .../technical-writing/tasks/incident-reply.md | 65 ---- .../technical-writing/tasks/pr-description.md | 42 --- .../tasks/reference-summary.md | 54 --- eval/technical-writing/tasks/runbook.md | 44 --- eval/technical-writing/tasks/tradeoff.md | 61 ---- 17 files changed, 1620 deletions(-) delete mode 100644 eval/technical-writing/.gitignore delete mode 100644 eval/technical-writing/FINDINGS.md delete mode 100644 eval/technical-writing/README.md delete mode 100644 eval/technical-writing/arms/baseline.txt delete mode 100644 eval/technical-writing/arms/oneliner.txt delete mode 100644 eval/technical-writing/arms/orwell.txt delete mode 100644 eval/technical-writing/arms/skill-v0.txt delete mode 100644 eval/technical-writing/arms/skill.txt delete mode 100644 eval/technical-writing/judge.py delete mode 100755 eval/technical-writing/run.sh delete mode 100755 eval/technical-writing/score.py delete mode 100644 eval/technical-writing/tasks/code-comment.md delete mode 100644 eval/technical-writing/tasks/incident-reply.md delete mode 100644 eval/technical-writing/tasks/pr-description.md delete mode 100644 eval/technical-writing/tasks/reference-summary.md delete mode 100644 eval/technical-writing/tasks/runbook.md delete mode 100644 eval/technical-writing/tasks/tradeoff.md diff --git a/eval/technical-writing/.gitignore b/eval/technical-writing/.gitignore deleted file mode 100644 index 89f9ac0..0000000 --- a/eval/technical-writing/.gitignore +++ /dev/null @@ -1 +0,0 @@ -out/ diff --git a/eval/technical-writing/FINDINGS.md b/eval/technical-writing/FINDINGS.md deleted file mode 100644 index 4c9f96e..0000000 --- a/eval/technical-writing/FINDINGS.md +++ /dev/null @@ -1,181 +0,0 @@ -# Findings — run 1 - -- Model: `sonnet`, 6 tasks x 4 arms = 24 generations, one per cell -- Judge: `opus`, 18 blinded pairs, every other arm against `skill` -- Skill revision: `65cf5b0` -- Clean room: no. See "Known confound" in README.md. - -## Result - -The skill wins the task it was designed for and loses everywhere else. - -| Task | Judge verdict vs `skill` | Deterministic failures (`skill` / others) | -| ------------------- | ------------------------ | ----------------------------------------- | -| `incident-reply` | **skill 3–0** | 0 / 1, 0, 0 | -| `code-comment` | skill 2–1 | 0 / 0, 0, 0 | -| `pr-description` | skill 0–2 | 0 / 0, 0, 0 | -| `reference-summary` | skill 0–3 | **0 / 1, 1, 1** | -| `runbook` | skill 0–3 | **0 / 0, 1, 1** | -| `tradeoff` | skill 0–3 | 0 / 0, 0, 0 | -| **total** | **skill 5–12** | 0 / 2, 2, 2 | - -## H1 — any instruction beats none: partly - -Sentence length improved. `orwell` took the best median at 13.2 words against -`baseline`'s 17.0, for about 120 tokens. - -Two results went the other way: - -- The skill has the field's **worst** nominalization rate (28.3 per 1000 words against - `baseline`'s 22.3) and worst agentless passive rate (5.1 against 4.1), despite gear 3 - telling it to use concrete verbs and explicit subjects. Those two rules did not take. -- On `runbook`, `oneliner` and `orwell` both **dropped the recovery path** — the "PUT - the old kid back as active, then retry" instruction. `baseline` kept it. Instructing - for brevity destroyed information that no instruction at all preserved. This is - ASD-STE100 rule 4.2 (no shortening by omission) observed rather than argued. - -## H2 — only the skill holds claim strength: yes, and only the judge can see it - -On `incident-reply` the skill won all three pairs, and the judge named the failures: - -- `baseline` "converts an explicitly unconfirmed root-cause hypothesis into a claim - that the team 'identified the issue,' and adds internal infrastructure detail" -- `orwell` "claims the issue was identified and fully resolved when the notes - explicitly mark the cause unconfirmed and leave 41 records unreconciled" -- `oneliner` "softens the rollback's causal role and exposes internal cluster detail" - -**The deterministic scorer found none of this.** It scored all four arms clean on that -task, because every arm contained a hedge word somewhere and regex cannot tell a hedge -attached to the cause from a hedge attached to anything else. Any future run should -treat the deterministic fidelity column as a smoke test and the judge as the instrument. - -## The finding that matters: the skill fabricates more than the arms without it - -The skill lost 12 pairs, and across 9 of them the judge gave the same reason. Three -fabrications, each verified against the task input by hand: - -| Output | Invented | Source says | -| ------------------------ | ------------------------------------------------- | ---------------------------------------------------------------------- | -| `tradeoff__skill` | `**Date:** 2026-08-03` | no date anywhere in the input | -| `reference-summary__skill` | "Enforce exactly-once delivery" | a dedupe table and a never-send-twice constraint, which is not a guarantee | -| `runbook__skill` | "If it is not present, add it now." | recovery is "PUT the old kid back as active, then retry" | - -Each one violates the skill's own top-priority rule — *add nothing the source did not -contain: no internal detail, no commitment, no date* — and the third invents a -procedure that **conflicts** with the documented recovery path, which is the most -dangerous class of error in a runbook. - -The fabricated date is worth noting twice: `2026-08-03` was the date in the operator's -contaminating context, not in the task. The confound is not cosmetic; it supplied -material the model then asserted. - -### Diagnosis - -The skill's completeness rules and its claim-safety rule pull in opposite directions, -and completeness was winning: - -- gear 3: "Name each error state and its consequence" -- gear 4: "Use these fields where they apply", "Always include a `Sources` field" - -Each is an instruction to produce a slot. With no source data for the slot, filling it -means inventing. The arms without those rules had nothing pushing them to fill -anything, so they invented less. The skill's one deterministic win — being the only arm -to emit a `Sources` field — comes from the same pressure that produced its losses. - -Notably the skill handled `Sources` itself honestly: *"Verbal/thread report only — no -code, config, or ticket references provided. All facts above are unverified."* The rule -works when the skill is told what to do about an empty field, and fails when it is not. - -### Applied - -Claim safety now states that it outranks every structural rule, not just concision, and -adds: leave a field empty and write `Unknown:` rather than fill it. The three -completeness rules were reworded to ask the writer to *look for* a fact rather than -supply one. Not yet re-run — the next run tests whether this closes the gap. - -## Limits - -- One generation per cell, one model, one judge. The fabrication finding rests on 9 - concurring judgments and 3 hand-verified instances; the style rates rest on n=1. -- The scorer had four bugs, all found by reading outputs behind findings that fired on - every arm. Assume more remain. A rule that flags every arm is measuring the rule. -- Hedge markers are now permissive enough that `claim_hardened` is effectively - unmeasured deterministically. That is why H2 needed the judge. -- `opus` judged with a fidelity-dominant rubric it was handed. A rubric that ranked - style first would likely reverse several verdicts. -- Nothing here measures whether the skill fires, or whether a gear survives past one - turn. - ---- - -# Findings — run 2 - -- Same 6 tasks. Five arms: `skill` is now the patched revision (`3555d06`, 137 lines) and - `skill-v0` is the original as submitted (`429617e`, 231 lines). -- `baseline`, `oneliner`, and `orwell` outputs were **not** regenerated, so both skill - versions face an identical competitor set. `skill` vs `skill-v0` is therefore a paired - comparison on fixed opponents. -- Judge: `opus`, 24 blinded pairs, every arm against `skill`. - -## Result: the patch helped, and the skill is now exactly average - -| Arm | wins | losses | -| ---------- | ---- | ------ | -| `skill` | 12 | 12 | -| `baseline` | 3 | 3 | -| `oneliner` | 3 | 3 | -| `orwell` | 3 | 3 | -| `skill-v0` | 3 | 3 | - -Run 1 had `skill` at 5–12. The patch moved it to 12–12 — a real gain, and also a dead -heat with every arm including the one-line prompt. - -**`skill` vs `skill-v0` is 3–3.** The 100-line refactor is a wash on judged quality. - -## The refactor lost style adherence - -| Arm | over 20 words | median sentence | agentless passive | nominalizations | -| ---------- | ------------- | --------------- | ----------------- | --------------- | -| `skill-v0` | **18.5%** | **11.8** | **4.0** | 21.0 | -| `orwell` | 29.9% | 13.2 | 4.6 | 23.0 | -| `skill` | 33.1% | 15.0 | 6.1 | **19.3** | -| `baseline` | 39.9% | 17.0 | 4.1 | 22.3 | - -The original produces the tightest prose in the field — better than the refactor on -every column but one, and better than Orwell. Plausible cause: the original restated its -style rules across the mode lists and the anti-pattern section, and pruning that -repetition as duplication also removed reinforcement. Tokens bought adherence. - -## The fabrication fix: one of three - -| Fabrication | `skill` (patched) | `skill-v0` | -| ------------------------------------------ | ----------------- | ---------- | -| runbook: invented "add it now" recovery | fixed | absent | -| reference-summary: "exactly-once delivery" | **still present** | **present** | -| tradeoff: invented `Date: 2026-08-03` | **still present** | absent | - -Both skill versions write *"Enforce exactly-once delivery per (notification_id, channel) -pair."* The source says only that the service must never send twice and keeps a dedupe -table. So this is not the completeness pressure diagnosed in run 1 — the original has no -mandatory-field rule and produces it anyway. - -**Revised diagnosis for that one.** Gear 3 says *keep precise software terms*. The -canonical term for the property being described is "exactly-once delivery", and reaching -for it is exactly what that rule asks for. But a canonical term carries the guarantees -the term implies, and this system only attempts the property through a dedupe table. -Vocabulary precision and claim strength can pull against each other, and gear 3 currently -only pushes one way. - -Candidate rule, not yet applied: *a term that names a guarantee asserts that guarantee. -Where the source describes a mechanism rather than proves a property, use the source's -own words.* - -The invented date appears only in the patched version. With n=1 per cell there is no way -to tell a regression from sampling noise. - -## Stopping the patch loop here - -Two patch cycles against 6 tasks at n=1 is the point where tuning becomes overfitting to -this task set. Further changes should wait on a defensible sample size — multiple -generations per cell, seed variance reported, and paired CIs — rather than another round -of chasing individual verdicts. diff --git a/eval/technical-writing/README.md b/eval/technical-writing/README.md deleted file mode 100644 index 3217b63..0000000 --- a/eval/technical-writing/README.md +++ /dev/null @@ -1,136 +0,0 @@ -# Technical-writing eval - -Compares four ways to ask a model for precise technical prose, on tasks built to -punish the specific failures each approach claims to fix. - -## The question - -Hacker News made a falsifiable claim about skills like `precise-technical-writing` -([thread](https://news.ycombinator.com/item?id=49114639)): - -> STE is part of the training set, so the skill is redundant and only pollutes your -> context window. — `lab14` - -> Seems to be doing too much, a 1 line in the system prompt is all you need. — `hsaliak` - -That is testable. So is the counter-claim: that the skill earns its tokens through -two things a one-liner cannot carry — gear selection and claim safety. - -Three hypotheses: - -- **H1 — Any instruction beats none.** All three instructed arms lower ambiguity and - structural slop against `baseline`. Expected to hold; it is the sanity check. -- **H2 — Only the skill holds claim strength.** On `incident-reply` and `tradeoff`, - arms without claim-safety rules harden hedged claims, invent detail, or flatten - disagreement. This is the differentiator. If it fails, the skill is cruft. -- **H3 — The skill beats the one-liner by more than its token cost.** The skill is - ~1500 tokens against ~25. If the margin is small, `hsaliak` is right. - -An arm winning on brevity alone proves nothing. ASD-STE100 rule 4.2 forbids -shortening by omission, and the thread's own top exchange shows why: `handfuloflight` -tightened "make sure that your AWS credentials are correct" to "ensure AWS -credentials are correct", and `harshreality` pointed out the rewrite changed the -meaning. Length is therefore **reported but never scored**. - -## Arms - -| Arm | Cost | What it is | -| ---------- | --------- | ------------------------------------------------------ | -| `baseline` | 0 tokens | The task, no style instruction | -| `oneliner` | ~25 | `hsaliak`'s system-prompt line, verbatim from HN | -| `orwell` | ~120 | Orwell's six rules, which beat STE in one HN benchmark | -| `skill` | ~1500 | The full `SKILL.md` | - -`orwell` is in because `gillesjacobs` cited a [benchmark](https://youtu.be/uJblcC4lKYw) -where those six rules beat the STE skill on slop indicators at a fraction of the -tokens. Unverified, and cheap to include as a control. - -## Tasks - -Six tasks, each mapping to a gear the skill would select, and each carrying declared -**traps** — specific failures the scorer looks for by name. - -| Task | Gear | Trap it sets | -| ------------------ | ---- | ---------------------------------------------------- | -| `code-comment` | 3 | Abstract nouns, passive voice, restating the code | -| `pr-description` | 2 | Marketing tone, burying the risk | -| `runbook` | 3 | Action before condition, unnamed failure states | -| `reference-summary`| 4 | Prose instead of fields, omitting `Sources` | -| `incident-reply` | 2 | **Claim inflation** — see below | -| `tradeoff` | 1 | **Over-application** — flattening live disagreement | - -The last two carry the experiment. - -`incident-reply` reproduces the failure `atoav` found in the HN skill's own example -output: an agent told to "just simplify the language" added internal detail and a -customer-facing commitment that were nowhere in the source. Its input contains an -**unconfirmed** root cause, an internal service name, and no agreed fix date. An arm -that states the cause as fact, leaks the internal name, or promises a date has failed -in a way no amount of clean prose redeems. - -`tradeoff` tests the opposite error. Its input is genuine exploration with unresolved -disagreement between two engineers. Controlled language applied here destroys the -content. `baseline` and `orwell` have no mechanism to avoid this; `oneliner` actively -pushes into it. Only the skill has a gear that says stay in prose. - -## Running it - -```sh -cd eval/technical-writing -./run.sh # 24 generations: 6 tasks x 4 arms -./run.sh --model opus # default is sonnet -./run.sh --tasks incident-reply # single task -python3 score.py out/ # deterministic metrics -> out/scores.json -python3 score.py out/ --markdown # readable table -``` - -Then judge blind: - -```sh -python3 judge.py out/ --pairs # emits anonymized A/B pairs + judge.md rubric -``` - -`judge.py` strips arm labels and randomizes presentation order, so the judging model -cannot see which arm wrote what. Run the emitted prompts through any model, or a -second `claude -p` call, and paste verdicts back. - -## Known confound, unresolved - -**Every generation carries the operator's global `~/.claude/CLAUDE.md`.** - -`claude --bare` is the only mode that skips user-memory discovery, and it requires -`ANTHROPIC_API_KEY` — OAuth and keychain are never read in bare mode. On an OAuth-only -machine there is no clean-room path. Verified by asking each configuration whether its -context mentioned a string unique to the operator's global memory; every non-bare -configuration answered yes, including with `--system-prompt` and `--tools ""`. - -Consequence: **absolute** numbers are not clean-room and should not be quoted as such. -**Relative** comparisons stay valid, because the contamination is identical across all -four arms within a run. It is a constant, not a variable. - -To get a clean run, set `ANTHROPIC_API_KEY` and pass `--bare`: - -```sh -ANTHROPIC_API_KEY=sk-... ./run.sh --bare -``` - -`run.sh` adds `--bare` only when that variable is set, and records which mode produced -each run in `out/manifest.json`. - -## What this cannot tell you - -- **Whether the skill fires.** This measures output quality once loaded. Invocation - reliability is a separate experiment against the `description`. -- **Whether the style survives.** Every generation is one turn. `boardwaalk`'s drift - complaint — "models drift immediately" — needs a multi-turn design. -- **Anything about a different model.** Arms may reorder across models. Run the model - you actually use. -- **Ambiguity, directly.** The scorer measures proxies for it. Only the blind judge - reads for meaning, and it is a model, not a panel of tired mechanics. - -## Sources - -- `../../skills/precise-technical-writing/SKILL.md` — the arm under test -- `../../chatgpt-share-asd-ste-100-for.md` — where the gears and modes came from -- https://news.ycombinator.com/item?id=49114639 — the critiques being tested -- https://www.asd-ste100.org/ — the standard gear 3 is modeled on diff --git a/eval/technical-writing/arms/baseline.txt b/eval/technical-writing/arms/baseline.txt deleted file mode 100644 index e69de29..0000000 diff --git a/eval/technical-writing/arms/oneliner.txt b/eval/technical-writing/arms/oneliner.txt deleted file mode 100644 index 610ac05..0000000 --- a/eval/technical-writing/arms/oneliner.txt +++ /dev/null @@ -1 +0,0 @@ -Output tokens are precious, be succinct in your responses. Use ASD-STE100 simplified technical english diff --git a/eval/technical-writing/arms/orwell.txt b/eval/technical-writing/arms/orwell.txt deleted file mode 100644 index a7f4c05..0000000 --- a/eval/technical-writing/arms/orwell.txt +++ /dev/null @@ -1,8 +0,0 @@ -Follow Orwell's six rules of writing: - -1. Never use a metaphor, simile, or other figure of speech which you are used to seeing in print. -2. Never use a long word where a short one will do. -3. If it is possible to cut a word out, always cut it out. -4. Never use the passive where you can use the active. -5. Never use a foreign phrase, a scientific word, or a jargon word if you can think of an everyday English equivalent. -6. Break any of these rules sooner than say anything outright barbarous. diff --git a/eval/technical-writing/arms/skill-v0.txt b/eval/technical-writing/arms/skill-v0.txt deleted file mode 100644 index 6def750..0000000 --- a/eval/technical-writing/arms/skill-v0.txt +++ /dev/null @@ -1,231 +0,0 @@ ---- -name: concise-technical-writing -description: Use when creating, editing, or refining durable technical communication such as documentation, code comments, PR descriptions, issue bodies, review replies, handoff notes, architecture notes, runbooks, agent instructions, or technical explanations where clarity, concision, structure, and unambiguous wording matter. Apply this skill implicitly for final wording of durable artifacts. -author: - name: Scott Pfister - email: scott.pfister@7factor.io ---- - -# Concise Technical Writing - -Use this skill as a final communication pass for durable technical writing. - -The goal is precision: engineering communication that is clear, concise, explicit, and easy for humans and agents to reuse. - -## Default Behavior - -Use this skill when the output is likely to be saved, reviewed, reused, searched, pasted, committed, or used by another agent later. - -Examples: - -- Code comments -- README sections -- Architecture docs -- ADRs -- API docs -- PR descriptions -- Review replies -- Issue descriptions -- Handoff docs -- Task plans -- Runbooks -- Agent skills -- Project instructions - -For normal conversation, use the principles lightly. Keep replies natural. Apply the full refinement pass when the output is durable or precision matters. - -## Workflow - -Before finalizing durable technical writing: - -1. Identify the artifact type. -2. Identify the dominant communication intent. -3. Select a writing mode. -4. Apply section-level modes when a section has a different intent. -5. Refine for clarity, concision, and structure. -6. Preserve claim strength. Keep assumptions labeled as assumptions. -7. Check that the final text keeps the original meaning. - -## Intent Classifier - -Classify by intent first and artifact type second. - -- `instruct`: Tell someone what to do. -- `specify`: State requirements, contracts, rules, invariants, or acceptance criteria. -- `reference`: Help future lookup. -- `explain`: Help understanding. -- `justify`: Explain rationale, tradeoffs, or risk. -- `explore`: Think through unknowns or options. -- `respond`: Answer a person, especially in review or collaboration. - -## Modes - -### `auto` - -Default mode. Classify the artifact and intent, then choose the right mode. - -Use a dominant mode for the artifact. Override by section only when the section's intent clearly differs. - -### `engineering` - -Use for concise natural technical prose. - -Good for: - -- PR descriptions -- Explainers -- Review replies -- Design summaries -- Normal documentation -- Rationale that does not need a long narrative - -Rules: - -- Prefer short paragraphs. -- Remove filler and generic praise. -- Use specific nouns and verbs. -- Keep terminology consistent. -- State assumptions and limits. -- Separate summary, details, risks, and verification when useful. -- Keep a human tone when replying to people. - -### `controlled` - -Use controlled technical English inspired by ASD-STE100. Use software terminology instead of the official STE approved word list. - -Good for: - -- Procedures -- Code comments -- API docs -- Runbooks -- Acceptance criteria -- Requirements -- Contracts -- Invariants -- Implementation notes - -Rules: - -- Use active voice. -- Use explicit subjects. -- Put one action or claim in each sentence. -- State conditions before actions. -- Use the same term for the same concept. -- Prefer concrete verbs over abstract nouns. -- Remove filler, hedging, and marketing language. -- Keep sentences short when practical. -- Use ordered lists for procedures. -- Use bullets or tables for sets of facts. -- Separate facts, assumptions, recommendations, and rationale. -- Make error states and consequences explicit. - -Use software vocabulary. Keep precise terms even when they are outside aircraft-maintenance vocabulary. Prefer clear sentences over mechanical rule compliance. - -### `reference` - -Use for dense, predictable lookup material. - -Good for: - -- Architecture maps -- Module summaries -- Repo guides -- Handoff state -- Agent-facing memory -- Source indexes - -Optimize for agent retrieval first and human readability second. - -Rules: - -- Prefer stable headings and fields. -- Use sparse prose. -- Use explicit names, paths, commands, owners, states, and links. -- Group facts under predictable labels. -- Include sources when available. -- Do not hide important facts in paragraphs. - -Useful fields: - -- Purpose -- Responsibilities -- Inputs -- Outputs -- Dependencies -- Invariants -- Failure Modes -- Sources -- Verification -- Open Questions - -### `narrative` - -Use when nuance, exploration, persuasion, or historical context matters. - -Good for: - -- Design exploration -- Tradeoff discussion -- Strategy -- RFC discussion -- ADR rationale -- Persuasive review context - -Rules: - -- Keep the prose clear and concise, but allow more connective tissue. -- Preserve uncertainty and disagreement. -- Explain why options were accepted or rejected. -- Do not flatten tradeoffs into false certainty. -- Keep facts separate from opinions and recommendations. - -## Common Artifact Mapping - -- Code comment: usually `controlled`; use `engineering` only for rationale. -- PR description: usually `engineering`; use `controlled` for testing, rollout, and reviewer instructions. -- Review reply: usually `engineering`; use `controlled` for exact commitments or steps. -- Runbook: usually `controlled`; use `engineering` for background. -- Architecture index: usually `reference`; use `engineering` for short context. -- ADR: mixed. Decision and consequences use `controlled`; rationale uses `engineering`; exploration uses `narrative`. -- Handoff doc: usually `reference`; use `controlled` for next steps and commands. -- Agent skill: usually `controlled` for procedure; use `reference` for lookup tables; use `engineering` for short context. - -## Claim Safety - -Concise writing must not overstate certainty. - -- Do not strengthen claims during refinement. -- Preserve uncertainty when the source is uncertain. -- Mark assumptions explicitly. -- Mark inferences explicitly when useful. -- Do not convert guesses into facts. -- Include source paths, commands, or evidence when the artifact is durable and evidence exists. -- If an important claim is unverified, label it as unverified or ask whether to verify it. - -Use clear labels when needed: - -- `Fact:` -- `Assumption:` -- `Inference:` -- `Recommendation:` -- `Unknown:` - -## Embedded Use Contract - -Other skills can depend on this skill with this compact instruction: - -> Before finalizing durable technical writing, apply `concise-technical-writing`: classify intent, select a mode, refine for clarity and concision, preserve claim strength, and optimize structure for later retrieval when applicable. - -## Anti-Patterns - -Avoid: - -- Applying controlled mode to brainstorming or early design exploration. -- Making human replies sound like maintenance procedures. -- Removing useful nuance from rationale. -- Replacing precise software terminology with generic words. -- Hiding assumptions to make the text shorter. -- Turning every artifact into a prose essay. -- Turning every artifact into a rigid template. -- Adding headings when a short answer is enough. diff --git a/eval/technical-writing/arms/skill.txt b/eval/technical-writing/arms/skill.txt deleted file mode 100644 index e4742ab..0000000 --- a/eval/technical-writing/arms/skill.txt +++ /dev/null @@ -1,137 +0,0 @@ ---- -name: precise-technical-writing -description: Use when writing or refining durable technical text — docs, code comments, PR descriptions, issue bodies, runbooks, handoff notes, agent instructions — or when another skill needs a final wording pass. Applies implicitly to durable artifacts. -metadata: - author: Scott Pfister (scott.pfister@7factor.io) ---- - -# Precise Technical Writing - -Write for precision: an engineer or agent reading this later must not have to guess what it meant. - -## Gears - -Control is a dial, not a switch. The four gears are one style at four compression ratios, ordered from most prose to least. - -| Gear | Name | Prose | Shift here when | -| ---- | ------------- | ----------- | -------------------- | -| 1 | `narrative` | Most | Exploration matters | -| 2 | `engineering` | Default | — | -| 3 | `controlled` | Little | Precision matters | -| 4 | `reference` | Almost none | Later lookup matters | - -Start in gear 2. Shift to 3 or 4 when precision or lookup matters. Drop to gear 1 only when exploration, persuasion, or live disagreement matters. - -Shift per section, not only per document. An ADR runs gear 3 for the decision, gear 2 for the rationale, gear 1 for the discussion. - -In conversation, keep the reply natural and apply the gear's spirit. Full refinement is for durable text. - -## Choosing a gear - -Classify intent first, artifact second. - -| Intent | Gear | -| --------------------------------------------------------------- | ---- | -| `instruct` — tell someone what to do | 3 | -| `specify` — state requirements, contracts, invariants, criteria | 3 | -| `look-up` — help someone find a fact later | 4 | -| `explain` — help someone understand | 2 | -| `justify` — give rationale, tradeoffs, or risk | 2 | -| `respond` — answer a person, in review or collaboration | 2 | -| `explore` — think through unknowns or options | 1 | - -When intent is mixed or unclear, fall back to the artifact: - -| Artifact | Gear | Shift for | -| ----------------------------------------------- | ------------------ | -------------------------------------- | -| Code comment | 3 | 2 for rationale | -| API doc, runbook, procedure, acceptance criteria | 3 | 2 for background | -| Agent skill, project instructions | 3 | 4 for lookup tables, 2 for context | -| PR description | 2 | 3 for testing, rollout, reviewer steps | -| Review reply | 2 | 3 for exact commitments | -| Explainer, design summary, issue body | 2 | — | -| Architecture index, module summary, repo guide | 4 | 2 for short context | -| Handoff note | 4 | 3 for next steps and commands | -| ADR | 3 for the decision | 2 for rationale, 1 for discussion | -| Brainstorm, strategy, RFC discussion | 1 | — | - -## Gear rules - -Each gear adds only what is listed here. - -### 1 `narrative` - -- Preserve uncertainty and disagreement. -- Say why each option was accepted or rejected. -- Leave tradeoffs as tradeoffs. -- Label facts, opinions, and recommendations separately. - -### 2 `engineering` - -- Give each paragraph one purpose. -- Cut filler, hedging, and marketing language. -- Name the assumptions and the limits. -- Split summary, detail, risk, and verification when the reader needs them apart. - -### 3 `controlled` - -Controlled English modeled on ASD-STE100, with software vocabulary in place of the approved word list. ASD-STE100 exists to remove ambiguity for readers who are not native English speakers. Write for that reader. - -- Use active voice and an explicit subject. -- Put one action or one claim in each sentence. -- State the condition before the action. -- Keep sentences under about 20 words. -- Use the same term for the same concept every time. -- Use concrete verbs in place of abstract nouns. -- Use ordered lists for procedures. -- Name each error state the source describes, and its consequence. -- Keep precise software terms. A clear sentence beats rule compliance. - -### 4 `reference` - -Built for an agent to retrieve first and a human to read second. - -- Use stable headings and field names. -- Put facts under predictable labels, where a reader finds them without reading prose. -- Give explicit names, paths, commands, owners, states, and links. -- Use these fields where the source supports them, and omit the rest: Purpose, Responsibilities, Inputs, Outputs, Dependencies, Invariants, Failure Modes, Open Questions. -- Always include a `Sources` field listing the paths the facts came from. A reference doc describes something it is not connected to, so the reader needs the path to re-check it. Without one, the doc drifts and nobody can tell. Where the source gives no paths, say that instead. - -## Claim safety - -Tightening the wording must not tighten the certainty. This rule outranks concision. - -- Carry each claim across at its original strength. -- Label assumptions as assumptions and unknowns as unknowns. -- Cite the source path, command, or evidence for each claim in durable text. -- Say a claim is unverified, or ask to verify it, rather than writing around it. -- Add nothing the source did not contain: no internal detail, no commitment, no date. -- Leave a field, step, or heading empty when the source does not support it. Write - `Unknown:` and stop. An empty field costs a reader nothing; an invented one costs - them a wrong decision. - -Use labels where the distinction carries weight: `Fact:` `Assumption:` `Unknown:` - -This rule also outranks every structural rule below it. A gear that asks for a field, -an error state, or a recovery path is asking you to look for one, never to supply one. - -Done when every claim in the output traces to a claim in the input at equal or weaker strength. - -## Refining text that already exists - -A rewrite keys off the prose it reads, so vocabulary changes and weak structure survives. Rebuild instead: - -1. Extract the claims, steps, and open questions as a bare list. -2. Pick the gear from that list, not from the old prose. -3. Write from the list. -4. Check the claim-safety criterion against the original. - -## Embedded use contract - -Other skills reach this skill with: - -> Before finalizing durable technical writing, apply `precise-technical-writing`: pick a gear, write from claims, hold claim strength steady, and structure for later retrieval. - -## Drift - -A gear holds for a few turns, then slips. Where a repo needs the style enforced instead of requested, gate on a prose linter such as [Vale](https://vale.sh) at pre-commit or `PostToolUse`. The skill sets the target; the gate holds it. diff --git a/eval/technical-writing/judge.py b/eval/technical-writing/judge.py deleted file mode 100644 index aec39a3..0000000 --- a/eval/technical-writing/judge.py +++ /dev/null @@ -1,192 +0,0 @@ -#!/usr/bin/env python3 -"""Blind pairwise judging for the technical-writing eval. - -The deterministic scorer measures proxies. Only a reader can judge whether an output -is actually unambiguous, so this builds pairwise comparisons a model can judge without -seeing which arm produced what. - -Blinding, because both biases are real and both would favor the skill: - - arm labels are stripped - - presentation order is shuffled with a fixed seed, so runs are reproducible - - the mapping is written to pairs/key.json, which the judge never reads - -Judged on the criteria the eval is actually about, with fidelity dominant. -""" -import argparse -import itertools -import json -import re -import sys -import random -from collections import defaultdict -from pathlib import Path - -RUBRIC = """You are comparing two versions of the same technical artifact, written from -the same source notes. You do not know who or what wrote either one. - -The source notes are given first. Read them carefully: several claims in them are -explicitly uncertain, and some details are internal. - -Judge in this order. Earlier criteria dominate later ones. - -1. FIDELITY (dominant). Does the version assert anything the notes do not support? - Does it turn an uncertain claim into a confident one? Does it invent a date, - a cause, or a commitment? Does it leak an internal name into text meant to be - external? Does it flatten a disagreement into a decision? A single fidelity - failure outweighs any amount of stylistic polish. - -2. AMBIGUITY. Could a competent engineer reading this at 3am act on it and get it - wrong? Look for unclear referents, missing actors, conditions stated after the - actions they govern, and unnamed failure states. - -3. RETRIEVABILITY. If someone needs one fact from this in six months, can they find - it without reading the whole thing? - -4. REGISTER. Is it free of marketing tone, filler openers, and signposting that - carries no information? - -Explicitly NOT criteria: - - Length. Shorter is not better. Splitting one complex sentence into three simple - ones is usually an improvement even though it adds words. - - Confidence of tone. A version that says "we have not confirmed this" is better - than one that sounds authoritative, if the notes did not confirm it. - -Respond as JSON only: - -{"winner": "A" | "B" | "tie", - "fidelity_failures": {"A": ["..."], "B": ["..."]}, - "reason": "one or two sentences", - "confidence": "high" | "medium" | "low"} -""" - - -def load_task_input(here: Path, task_id: str) -> str: - raw = (here / "tasks" / f"{task_id}.md").read_text().split("---", 2) - return raw[2].strip() - - -def main() -> None: - ap = argparse.ArgumentParser() - ap.add_argument("out_dir", type=Path) - ap.add_argument("--seed", type=int, default=20260803) - ap.add_argument("--baseline-arm", default=None, - help="compare every arm against this one instead of all pairs") - ap.add_argument("--resolve", type=Path, default=None, - help="de-blind a verdicts file produced by the judging loop") - args = ap.parse_args() - - here = Path(__file__).parent - rng = random.Random(args.seed) - - if args.resolve: - resolve(args.out_dir, args.resolve) - return - - outputs = {} - for f in sorted(args.out_dir.glob("*__*.txt")): - task_id, arm = f.stem.split("__", 1) - outputs.setdefault(task_id, {})[arm] = f.read_text().strip() - - pairs_dir = args.out_dir / "pairs" - pairs_dir.mkdir(exist_ok=True) - for stale in pairs_dir.glob("*"): - stale.unlink() - - key = {} - n = 0 - for task_id, by_arm in sorted(outputs.items()): - arms = sorted(by_arm) - combos = ([(args.baseline_arm, a) for a in arms if a != args.baseline_arm] - if args.baseline_arm else list(itertools.combinations(arms, 2))) - for left, right in combos: - if left not in by_arm or right not in by_arm: - continue - shown = [(left, by_arm[left]), (right, by_arm[right])] - rng.shuffle(shown) - pair_id = f"{task_id}__{n:03d}" - key[pair_id] = {"task": task_id, "A": shown[0][0], "B": shown[1][0]} - (pairs_dir / f"{pair_id}.txt").write_text( - f"{RUBRIC}\n" - f"=== SOURCE NOTES ===\n\n{load_task_input(here, task_id)}\n\n" - f"=== VERSION A ===\n\n{shown[0][1]}\n\n" - f"=== VERSION B ===\n\n{shown[1][1]}\n" - ) - n += 1 - - (pairs_dir / "key.json").write_text(json.dumps(key, indent=2)) - print(f"wrote {n} blinded pairs to {pairs_dir}") - print(f"mapping in {pairs_dir / 'key.json'} — do not feed it to the judge\n") - print("judge them with (prompt on stdin — --tools is variadic and would eat it):") - print(f""" - for p in {pairs_dir}/*__*.txt; do - echo "=== $(basename "$p" .txt)" - ( cd "$(mktemp -d)" && claude -p --model opus --tools "" \\ - --no-session-persistence < "$p" ) - done | tee {args.out_dir}/verdicts.txt -""") - print(f"then resolve labels: python3 {Path(__file__).name} " - f"{args.out_dir} --resolve {args.out_dir}/verdicts.txt") - - -def resolve(out_dir: Path, verdicts_file: Path) -> None: - """Map blinded A/B verdicts back to arm names and tally wins.""" - key = json.loads((out_dir / "pairs" / "key.json").read_text()) - text = verdicts_file.read_text() - - wins = defaultdict(int) - losses = defaultdict(int) - ties = defaultdict(int) - rows = [] - - # Blocks are delimited by the "=== " lines the judging loop echoes. - blocks = re.split(r"^===\s*(\S+)\s*$", text, flags=re.M)[1:] - for pair_id, block in zip(blocks[::2], blocks[1::2]): - if pair_id not in key: - print(f"warn: verdict for unknown pair {pair_id}", file=sys.stderr) - continue - m = re.search(r"\{.*\}", block, re.S) - if not m: - print(f"warn: no JSON verdict in block {pair_id}", file=sys.stderr) - continue - try: - v = json.loads(m.group(0)) - except json.JSONDecodeError: - print(f"warn: unparseable verdict in block {pair_id}", file=sys.stderr) - continue - - mapping = key[pair_id] - winner = v.get("winner") - if winner == "tie": - ties[mapping["A"]] += 1 - ties[mapping["B"]] += 1 - won, lost = None, None - elif winner in ("A", "B"): - won = mapping[winner] - lost = mapping["B" if winner == "A" else "A"] - wins[won] += 1 - losses[lost] += 1 - else: - print(f"warn: verdict {winner!r} in {pair_id} is not A/B/tie", file=sys.stderr) - continue - - rows.append({ - "pair": pair_id, "task": mapping["task"], - "winner": won, "loser": lost, - "confidence": v.get("confidence"), "reason": v.get("reason"), - "fidelity_failures": {mapping[k]: v.get("fidelity_failures", {}).get(k, []) - for k in ("A", "B")}, - }) - - dest = out_dir / "verdicts.json" - dest.write_text(json.dumps(rows, indent=2)) - - arms = sorted(set(list(wins) + list(losses) + list(ties))) - print("| Arm | wins | losses | ties |") - print("|---|---|---|---|") - for a in arms: - print(f"| `{a}` | {wins[a]} | {losses[a]} | {ties[a]} |") - print(f"\njudged {len(rows)} pairs; detail in {dest}") - - -if __name__ == "__main__": - main() diff --git a/eval/technical-writing/run.sh b/eval/technical-writing/run.sh deleted file mode 100755 index 930b616..0000000 --- a/eval/technical-writing/run.sh +++ /dev/null @@ -1,107 +0,0 @@ -#!/usr/bin/env bash -# Generate one output per (task, arm) pair via headless claude. -# -# Runs from a scratch cwd so no project CLAUDE.md is discovered. The operator's -# global ~/.claude/CLAUDE.md still loads unless --bare is available; see README.md -# under "Known confound". Mode is recorded in out/manifest.json. -set -euo pipefail - -HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" -MODEL="sonnet" -OUT="$HERE/out" -ONLY_TASKS="" -ONLY_ARMS="" -BARE="" - -while [[ $# -gt 0 ]]; do - case "$1" in - --model) MODEL="$2"; shift 2 ;; - --tasks) ONLY_TASKS="$2"; shift 2 ;; - --arms) ONLY_ARMS="$2"; shift 2 ;; - --out) OUT="$2"; shift 2 ;; - --bare) BARE="1"; shift ;; - *) echo "unknown option: $1" >&2; exit 2 ;; - esac -done - -if [[ -n "$BARE" && -z "${ANTHROPIC_API_KEY:-}" ]]; then - echo "--bare requires ANTHROPIC_API_KEY (bare mode never reads OAuth or keychain)." >&2 - exit 2 -fi - -SCRATCH="$(mktemp -d)" -trap 'rm -rf "$SCRATCH"' EXIT -mkdir -p "$OUT" - -matches() { [[ -z "$2" ]] || [[ ",$2," == *",$1,"* ]]; } - -count=0 -for task_file in "$HERE"/tasks/*.md; do - task="$(basename "$task_file" .md)" - matches "$task" "$ONLY_TASKS" || continue - - # Split the task file: YAML frontmatter is config, everything after is the prompt. - brief="$(python3 -c " -import sys, yaml -raw = open(sys.argv[1]).read().split('---', 2) -meta = yaml.safe_load(raw[1]) -print(meta['brief'].strip()) -" "$task_file")" - body="$(python3 -c " -import sys -print(open(sys.argv[1]).read().split('---', 2)[2].strip()) -" "$task_file")" - - for arm_file in "$HERE"/arms/*.txt; do - arm="$(basename "$arm_file" .txt)" - matches "$arm" "$ONLY_ARMS" || continue - - dest="$OUT/${task}__${arm}.txt" - if [[ -s "$dest" ]]; then - echo "skip $task / $arm (exists)" - continue - fi - - prompt="$brief - -$body - -Output only the finished artifact. No preamble, no explanation of your choices." - - sys="$(cat "$arm_file")" - # The prompt goes in on stdin, never as a positional argument: --tools is - # variadic, so `--tools "" "$prompt"` silently swallows the prompt. That bug - # hit only the baseline arm, whose flag list ends with --tools. - cmd=(claude -p --model "$MODEL" --no-session-persistence --tools "") - [[ -n "$BARE" ]] && cmd+=(--bare) - # An empty system prompt would be rejected; baseline gets no flag at all. - [[ -n "$sys" ]] && cmd+=(--append-system-prompt "$sys") - - echo "gen $task / $arm" - ( cd "$SCRATCH" && printf '%s' "$prompt" | "${cmd[@]}" ) > "$dest" || { - echo "FAILED $task / $arm" >&2 - rm -f "$dest" - continue - } - count=$((count + 1)) - done -done - -python3 - "$OUT" "$MODEL" "${BARE:-0}" <<'PY' -import json, os, subprocess, sys -out, model, bare = sys.argv[1], sys.argv[2], sys.argv[3] == "1" -rev = subprocess.run(["git", "rev-parse", "--short", "HEAD"], - capture_output=True, text=True).stdout.strip() -json.dump({ - "model": model, - "bare": bare, - "clean_room": bare, - "skill_revision": rev, - "note": ("clean room" if bare else - "operator global CLAUDE.md present in all arms; relative comparisons only"), - "outputs": sorted(f for f in os.listdir(out) if f.endswith(".txt")), -}, open(os.path.join(out, "manifest.json"), "w"), indent=2) -PY - -echo "generated $count new output(s) into $OUT" -echo "next: python3 $HERE/score.py $OUT --markdown" diff --git a/eval/technical-writing/score.py b/eval/technical-writing/score.py deleted file mode 100755 index 21091de..0000000 --- a/eval/technical-writing/score.py +++ /dev/null @@ -1,318 +0,0 @@ -#!/usr/bin/env python3 -"""Deterministic scoring for the technical-writing eval. - -Two metric families, deliberately separated: - - STYLE proxies for ambiguity and slop. Every instructed arm should improve these. - FIDELITY whether the arm invented, hardened, or leaked a claim. Only the skill has - rules aimed at these, so this is where H2 is decided. - -Length is reported and never scored. ASD-STE100 rule 4.2 forbids shortening by -omission, so a shorter output is not a better one. - -A FIDELITY failure is not tradeable against a STYLE win: the two are reported -separately and never summed into one number. -""" -import argparse -import json -import re -import sys -from collections import defaultdict -from pathlib import Path - -import yaml - -# Slop markers. Sourced from the HN thread's own complaints (heavy signposting, -# meta-commentary, marketing register) plus the usual AI tells. -TELLS = [ - r"\bit'?s worth noting\b", r"\bit'?s important to (note|remember)\b", - r"\bthat said\b", r"\bat the end of the day\b", r"\bin today'?s\b", - r"\bdelve into\b", r"\bnavigat(e|ing) the\b", r"\bunlock(s|ing)?\b", - r"\bseamless(ly)?\b", r"\brobust\b", r"\bcomprehensive\b", r"\bleverag(e|ing)\b", - r"\butili[sz](e|ing)\b", r"\bfacilitat(e|ing)\b", r"\borchestrat(e|ing)\b", - r"\bstreamlin(e|ing)\b", r"\bcutting[- ]edge\b", r"\bbest practices?\b", - r"\bgame[- ]chang(er|ing)\b", r"\bplays? a (key|vital|crucial|central) role\b", - r"\bcritical component\b", r"\bhere'?s (the|what|why|how)\b", - r"\bthe (key|real) (insight|takeaway|question) (is|here)\b", - r"\bnot (just|only) \w+ (but|—)\b", r"\bmoving forward\b", -] - -HEDGES = [ - r"\bmight\b", r"\bperhaps\b", r"\barguably\b", r"\bsomewhat\b", r"\bfairly\b", - r"\bquite\b", r"\brelatively\b", r"\bgenerally\b", r"\btypically\b", - r"\busually\b", r"\bin some cases\b", r"\bcould potentially\b", r"\bit seems\b", -] - -# Passive voice with no named actor: "is performed", "was updated" not followed by "by". -PASSIVE = re.compile( - r"\b(?:is|are|was|were|be|been|being)\s+(\w+(?:ed|en))\b(?!\s+by\b)", re.I) - -# Abstract nouns standing where a verb belongs. -NOMINALIZATION = re.compile( - r"\b\w{4,}(?:tion|sion|ment|ance|ence|ity|ness)\b", re.I) - -CODE_FENCE = re.compile(r"```.*?```", re.S) -INLINE_CODE = re.compile(r"`[^`]*`") - -# Markdown emphasis around a field name hid it from the `required` patterns: -# "**Sources:**" never matched /^#*\s*sources/. Normalize before matching. -EMPHASIS = re.compile(r"[*_]{1,3}") - - -def normalize(text: str) -> str: - """Strip markdown emphasis and list markers so field-name patterns can anchor.""" - lines = [] - for ln in text.splitlines(): - ln = EMPHASIS.sub("", ln) - ln = re.sub(r"^\s*[-*+]\s+", "", ln) - lines.append(ln.strip()) - return "\n".join(lines) - - -def prose_only(text: str) -> str: - """Strip code blocks. Style rules apply to prose, not to the code being documented.""" - return INLINE_CODE.sub(" ", CODE_FENCE.sub(" ", text)) - - -def sentences(text: str) -> list[str]: - # Skip list markers and headings; they are structure, not sentences. - lines = [ln for ln in text.splitlines() - if ln.strip() and not ln.lstrip().startswith("#")] - joined = " ".join(lines) - parts = re.split(r"(?<=[.!?])\s+(?=[A-Z(\[])", joined) - return [p.strip() for p in parts if len(p.split()) >= 3] - - -def count_patterns(text: str, patterns: list[str]) -> int: - return sum(len(re.findall(p, text, re.I)) for p in patterns) - - -def per_kw(n: int, words: int) -> float: - """Rate per 1000 words, so a longer output is not penalized for being longer.""" - return round(n * 1000 / words, 1) if words else 0.0 - - -def load_task(path: Path) -> dict: - raw = path.read_text().split("---", 2) - meta = yaml.safe_load(raw[1]) - meta["input"] = raw[2] - validate_task(path.stem, meta) - return meta - - -def validate_task(task_id: str, meta: dict) -> None: - """Reject rules that fire on every output regardless of arm. - - A term group holding both "pool" and "connection pool" always reports drift, - because matching the long form also matches the short one. Such a rule measures - the rule, not the arm. - """ - for group in meta.get("terms") or []: - low = [t.lower() for t in group] - for a in low: - for b in low: - if a != b and a in b: - raise SystemExit( - f"{task_id}: term group {group} has '{a}' inside '{b}', so it " - f"flags every output. Use terms that are not substrings.") - for rule in meta.get("must_hedge") or []: - if "presence" not in rule: - raise SystemExit( - f"{task_id}: must_hedge rule '{rule.get('claim')}' needs a 'presence' " - f"list, else an omitted claim is miscounted as a hardened one.") - - -def score_style(text: str) -> dict: - prose = prose_only(text) - words = len(prose.split()) - sents = sentences(prose) - lengths = sorted(len(s.split()) for s in sents) - - def pct(p): - return lengths[min(int(len(lengths) * p), len(lengths) - 1)] if lengths else 0 - - return { - "words": words, - "sentences": len(sents), - "median_sentence_words": pct(0.5), - "p90_sentence_words": pct(0.9), - "over_20_words_pct": round( - 100 * sum(1 for n in lengths if n > 20) / len(lengths), 1) if lengths else 0.0, - "tells_per_1k": per_kw(count_patterns(prose, TELLS), words), - "hedges_per_1k": per_kw(count_patterns(prose, HEDGES), words), - "agentless_passive_per_1k": per_kw(len(PASSIVE.findall(prose)), words), - "nominalizations_per_1k": per_kw(len(NOMINALIZATION.findall(prose)), words), - } - - -def score_fidelity(text: str, task: dict) -> dict: - """Did the arm invent, harden, leak, or flatten a claim? - - Every finding names the task rule it broke, so a failure is auditable rather - than a number to trust. - """ - findings = [] - norm = normalize(text) - - for rule in task.get("forbidden") or []: - hits = re.findall(rule["pattern"], norm, re.M) - if hits: - findings.append({ - "kind": "forbidden", - "why": rule["why"], - "matched": list({h if isinstance(h, str) else h[0] for h in hits})[:3], - }) - - for rule in task.get("must_hedge") or []: - # An absent claim has no hedge markers either, so absence and hardening look - # identical unless presence is tested first. They are opposite outcomes: - # hardening asserts something unsupported, omission just leaves it out. - present = any(re.search(rf"\b{re.escape(m)}", norm, re.I) - for m in rule["presence"]) - hedged = any(re.search(rf"\b{re.escape(m)}", norm, re.I) - for m in rule["markers"]) - if present and not hedged: - findings.append({ - "kind": "claim_hardened", - "claim": rule["claim"], - "why": rule["why"], - "matched": [], - }) - elif not present and not rule.get("absent_ok", False): - findings.append({ - "kind": "claim_absent", - "claim": rule["claim"], - "why": f"{rule['why']} (claim not raised at all)", - "matched": [], - }) - - for rule in task.get("required") or []: - if not re.search(rule["pattern"], norm, re.M): - findings.append({ - "kind": "omitted", - "why": rule["why"], - "matched": [], - }) - - drift = [] - for group in task.get("terms") or []: - used = [t for t in group if re.search(rf"\b{re.escape(t)}\b", text, re.I)] - if len(used) > 1: - drift.append(used) - if drift: - findings.append({ - "kind": "term_drift", - "why": "same concept named more than one way", - "matched": [" / ".join(g) for g in drift], - }) - - return { - "failures": len(findings), - "claim_hardened": sum(1 for f in findings if f["kind"] == "claim_hardened"), - "claim_absent": sum(1 for f in findings if f["kind"] == "claim_absent"), - "forbidden": sum(1 for f in findings if f["kind"] == "forbidden"), - "omitted": sum(1 for f in findings if f["kind"] == "omitted"), - "term_drift": sum(1 for f in findings if f["kind"] == "term_drift"), - "findings": findings, - } - - -def main() -> int: - ap = argparse.ArgumentParser() - ap.add_argument("out_dir", type=Path) - ap.add_argument("--markdown", action="store_true") - args = ap.parse_args() - - here = Path(__file__).parent - tasks = {p.stem: load_task(p) for p in sorted((here / "tasks").glob("*.md"))} - - results = {} - for f in sorted(args.out_dir.glob("*__*.txt")): - task_id, arm = f.stem.split("__", 1) - if task_id not in tasks: - print(f"warn: no task definition for {task_id}, skipping", file=sys.stderr) - continue - text = f.read_text() - # An in-flight generation truncates its destination before filling it, so a - # concurrent score run sees an empty file and reports every hedge as missing. - # Refuse rather than emit a plausible wrong number. - if len(text.split()) < 20: - print(f"warn: {f.name} has {len(text.split())} words — partial or failed " - f"generation, skipping", file=sys.stderr) - continue - results[f.stem] = { - "task": task_id, "arm": arm, - "style": score_style(text), - "fidelity": score_fidelity(text, tasks[task_id]), - } - - if not results: - print("no outputs found — run ./run.sh first", file=sys.stderr) - return 1 - - dest = args.out_dir / "scores.json" - dest.write_text(json.dumps(results, indent=2)) - - if args.markdown: - emit_markdown(results) - print(f"\nwrote {dest}", file=sys.stderr) - return 0 - - -def emit_markdown(results: dict) -> None: - arms = sorted({r["arm"] for r in results.values()}) - tasks = sorted({r["task"] for r in results.values()}) - - print("## Fidelity failures (lower is better; this decides H2)\n") - print("| Task | " + " | ".join(arms) + " |") - print("|---" * (len(arms) + 1) + "|") - for t in tasks: - row = [] - for a in arms: - r = results.get(f"{t}__{a}") - row.append("—" if not r else str(r["fidelity"]["failures"])) - print(f"| `{t}` | " + " | ".join(row) + " |") - totals = [] - for a in arms: - totals.append(str(sum(r["fidelity"]["failures"] - for r in results.values() if r["arm"] == a))) - print("| **total** | " + " | ".join(f"**{x}**" for x in totals) + " |") - - print("\n### Failures by kind\n") - print("| Arm | claim hardened | claim absent | forbidden | omitted | term drift |") - print("|---|---|---|---|---|---|") - for a in arms: - rs = [r for r in results.values() if r["arm"] == a] - print(f"| `{a}` | " - + " | ".join(str(sum(r["fidelity"][k] for r in rs)) - for k in ("claim_hardened", "claim_absent", "forbidden", - "omitted", "term_drift")) - + " |") - - print("\n## Style (rates per 1000 words; lower is better)\n") - keys = ["tells_per_1k", "hedges_per_1k", "agentless_passive_per_1k", - "nominalizations_per_1k", "over_20_words_pct", "median_sentence_words"] - print("| Arm | " + " | ".join(k.replace("_per_1k", "").replace("_", " ") - for k in keys) + " | words |") - print("|---" * (len(keys) + 2) + "|") - for a in arms: - rs = [r for r in results.values() if r["arm"] == a] - cells = [f"{sum(r['style'][k] for r in rs) / len(rs):.1f}" for k in keys] - wc = sum(r["style"]["words"] for r in rs) - print(f"| `{a}` | " + " | ".join(cells) + f" | {wc} |") - print("\n_Word counts are context, not score._") - - print("\n## Every finding\n") - for name in sorted(results): - fs = results[name]["fidelity"]["findings"] - if not fs: - continue - print(f"**`{name}`**\n") - for f in fs: - matched = f" — matched: `{'`, `'.join(f['matched'])}`" if f["matched"] else "" - print(f"- `{f['kind']}`: {f['why']}{matched}") - print() - - -if __name__ == "__main__": - sys.exit(main()) diff --git a/eval/technical-writing/tasks/code-comment.md b/eval/technical-writing/tasks/code-comment.md deleted file mode 100644 index 46062dc..0000000 --- a/eval/technical-writing/tasks/code-comment.md +++ /dev/null @@ -1,42 +0,0 @@ ---- -id: code-comment -gear: 3 -brief: > - Write the doc comment for this function. The reader is an engineer who has to call it - correctly and has not read its body. -terms: - - ["stale entries", "orphaned entries", "dangling entries"] - - ["entry", "record", "item"] -forbidden: - - pattern: "(?i)\\b(utili[sz]e|leverage|facilitate|orchestrat)" - why: abstract verb where a concrete one exists - - pattern: "(?i)\\b(is|are|be) (performed|executed|carried out|utili[sz]ed)\\b" - why: passive voice with no actor - - pattern: "(?i)\\bthis (function|method) (is responsible for|serves to|aims to)\\b" - why: filler opener -must_hedge: [] -required: - - pattern: "(?i)(raise|throw|error|exception|ValueError)" - why: the function has a failure mode the caller must handle; gear 3 requires naming it - - pattern: "(?i)(mutat|in place|modifi|changes the|removes from|deletes from)" - why: the function mutates its argument, which is the one thing a caller must know ---- - -## Input - -```python -def reconcile(cache, redis, *, ttl=None, dry_run=False): - if ttl is not None and ttl <= 0: - raise ValueError("ttl must be positive") - live = set(redis.scan_iter(match="entry:*")) - stale = [k for k in cache if f"entry:{k}" not in live] - if dry_run: - return stale - for k in stale: - del cache[k] - for k in live - {f"entry:{k}" for k in cache}: - cache[k.removeprefix("entry:")] = redis.get(k) - if ttl: - redis.expire(k, ttl) - return stale -``` diff --git a/eval/technical-writing/tasks/incident-reply.md b/eval/technical-writing/tasks/incident-reply.md deleted file mode 100644 index c512ade..0000000 --- a/eval/technical-writing/tasks/incident-reply.md +++ /dev/null @@ -1,65 +0,0 @@ ---- -id: incident-reply -gear: 2 -brief: > - Write the customer-facing status update for this incident, for posting on our public - status page. Use only what the internal notes support. -terms: - # "ingest" is excluded deliberately: it appears inside the forbidden internal - # hostname, so counting it here would double-charge the same mistake. - - ["outage", "incident", "disruption"] - - ["customer", "client", "user"] -forbidden: - - pattern: "(?i)\\bkafka-ingest-3\\b" - why: internal hostname, not for a public status page - - pattern: "(?i)\\bLegacyBlobShim\\b" - why: internal class name, not for a public status page - - pattern: "(?i)root cause (is|was)\\b" - why: states an unconfirmed cause as established fact - - pattern: "(?i)(this week|by (monday|tuesday|wednesday|thursday|friday)|within \\d+ (hours|days)|by end of)" - why: invents a delivery commitment the notes do not contain - - pattern: "(?i)(no data (was )?lost|nothing was lost|all data (is|was) (safe|intact))" - why: notes say the 41 records are still unaccounted for - - pattern: "(?i)(we (deeply )?apologi[sz]e for the inconvenience|we sincerely apologi[sz]e)" - why: boilerplate the notes do not ask for and gear 2 cuts as filler -must_hedge: - # absent_ok on both: for a public status page, saying nothing about an unconfirmed - # cause or about records the notes forbid calling lost is defensible restraint. - # The failure being measured is raising the claim and stating it confidently. - - claim: cause - presence: ["cause", "reason", "shim", "retry", "offset", "trigger"] - markers: ["likely", "appears", "suspect", "not confirmed", "unconfirmed", "possible", - "may", "believe", "preliminary", "investigat", "determin", "identif", - "not yet", "continue", "continuing"] - absent_ok: true - why: the notes mark the cause as unconfirmed - - claim: missing_records - presence: ["41", "unaccounted", "data loss", "lost", "missing"] - markers: ["unaccounted", "still", "not yet", "investigat", "unresolved", - "outstanding", "reconcil", "review"] - absent_ok: true - why: the 41 records are unresolved, not recovered ---- - -## Input - -Internal incident notes, verbatim from the on-call channel: - -``` -2026-07-29 14:02 — alerts firing on upload failures, ~8% of POST /v2/upload returning 500 -2026-07-29 14:20 — kafka-ingest-3 is the only broker showing the errors. Restarted it. - error rate drops to ~1% but doesn't clear. -2026-07-29 15:10 — sam thinks it's the LegacyBlobShim retry path double-acking and - dropping the offset. NOT CONFIRMED. we don't have the trace data to - prove it, the sampling was at 1% during the window. -2026-07-29 15:45 — rolled back to build 4471. error rate at 0% since 15:38. -2026-07-29 16:30 — reconciliation finds 41 upload records that we can't match to a - stored blob. could be the same bug, could be a reconciliation - artifact. still digging. do NOT tell anyone these are lost yet. -2026-07-30 09:15 — priya: we should fix the shim retry path but nobody has scoped it. - not on this sprint. no date. -2026-07-30 09:40 — 41 records still unaccounted for. reconciliation job rerun didn't - change the number. -``` - -Customers affected: uploads failing intermittently for roughly 100 minutes. diff --git a/eval/technical-writing/tasks/pr-description.md b/eval/technical-writing/tasks/pr-description.md deleted file mode 100644 index 7031fa7..0000000 --- a/eval/technical-writing/tasks/pr-description.md +++ /dev/null @@ -1,42 +0,0 @@ ---- -id: pr-description -gear: 2 -brief: > - Write the pull request description for this change. -terms: - - ["timeout", "deadline"] -forbidden: - - pattern: "(?i)\\b(significantly|dramatically|greatly|massively) (improv|reduc|enhanc|boost)" - why: unquantified intensifier where a number exists in the input - - pattern: "(?i)\\b(robust|seamless|comprehensive|elegant|clean) (solution|implementation|approach|fix)\\b" - why: marketing register - - pattern: "(?i)this (PR|change) (introduces|brings) (a|an) (new|improved)\\b" - why: filler opener -must_hedge: - - claim: unverified_gain - presence: ["40", "1.1", "p99", "faster", "improvement", "benchmark", "latency"] - markers: ["local", "not measured", "unverified", "staging", "expect", "should", - "have not", "no production", "laptop", "only", "unconfirmed"] - why: the 40% figure came from a local benchmark and has not been measured in production -required: - - pattern: "(?i)(migration|deploy|order|before|first|drain)" - why: the change has a deploy-ordering hazard that a reviewer must be told about ---- - -## Input - -What changed, from the author's own notes: - -``` -- swapped the per-request psycopg connection for a shared pool (pool size 20) -- added statement_timeout=5s, previously unbounded -- deleted the retry wrapper in db/legacy.py, the pool handles it now -- local benchmark: p99 on /reports went 1.9s -> 1.1s (about 40% better). - I have NOT measured this in staging or prod. Just my laptop, 200 iterations. -- CAREFUL: the pool reads DB_POOL_SIZE from env. If you deploy this before the - config change lands in the terraform repo, it defaults to 5 and reports will - queue. The terraform PR is separate and has to go first. -- statement_timeout will now kill two known-slow admin queries that used to - finish in ~8s. Those will start erroring. I think that's correct behavior but - someone from the admin team should confirm. -``` diff --git a/eval/technical-writing/tasks/reference-summary.md b/eval/technical-writing/tasks/reference-summary.md deleted file mode 100644 index bdec853..0000000 --- a/eval/technical-writing/tasks/reference-summary.md +++ /dev/null @@ -1,54 +0,0 @@ ---- -id: reference-summary -gear: 4 -brief: > - Write the reference entry for this service in our architecture index. Another agent - will read it to answer questions without reading the code. -terms: - - ["dedupe table", "dedupe log", "sent log"] -forbidden: - - pattern: "(?i)\\b(plays a (key|vital|central) role|is a critical component|serves as the backbone)\\b" - why: prose padding in a lookup artifact -must_hedge: - - claim: ownership_unknown - presence: ["owner", "Platform", "Growth", "catalog", "team"] - markers: ["unknown", "unclear", "not documented", "no owner", "unowned", "TBD", - "unassigned", "disputed", "contested", "never accepted"] - why: the input says ownership is disputed and undocumented -required: - - pattern: "(?i)^#*\\s*sources?\\b" - why: gear 4 requires a Sources field so the entry can be re-checked against the code - - pattern: "(?i)(invariant|must always|guarantee)" - why: the input contains an invariant, which is the highest-value field for an agent ---- - -## Input - -Whatever anyone could remember about the service, collected in a thread: - -``` -NotificationDispatcher. Lives in services/notify/. Entry point is -Dispatcher.RunAsync in Dispatcher.cs. - -It reads off the notifications topic and fans out to email (SendGrid), push (FCM), -and SMS (Twilio). Config for which channels are on per-tenant is in -NotifyOptions.cs, loaded from app config. - -Depends on: IdentityService (to resolve a user id to contact details), TenantConfig -(channel toggles), and the three vendor SDKs. - -Important thing nobody wrote down: it must never send the same notification twice -for the same (notification_id, channel) pair. There's a dedupe table, -notify_sent_log, and the whole design assumes that constraint holds. If you add a -channel you have to add it to the dedupe key or you get duplicate sends. - -Failure modes: SendGrid 429s a lot, there's a backoff. FCM token expiry produces a -permanent failure that gets logged and dropped, deliberately. Twilio failures retry -3x then dead-letter to notify_dlq. - -Who owns it: honestly unclear. It was the Platform team, then it moved to Growth -during the reorg, but Growth says they never accepted it. Nobody has updated the -service catalog. - -Open question: nobody knows if the dedupe table is ever pruned. It's 400M rows. -``` diff --git a/eval/technical-writing/tasks/runbook.md b/eval/technical-writing/tasks/runbook.md deleted file mode 100644 index 14141ca..0000000 --- a/eval/technical-writing/tasks/runbook.md +++ /dev/null @@ -1,44 +0,0 @@ ---- -id: runbook -gear: 3 -brief: > - Turn these notes into the runbook step for rotating the signing key. The reader is - on-call at 3am and has not done this before. -terms: - - ["verify set", "verify_kids", "verification list"] - - ["revoke", "invalidate"] -forbidden: - - pattern: "(?i)^\\s*\\d+\\..*\\bif (the|there|you)\\b.*," - why: condition placed after the action instead of before it - - pattern: "(?i)\\b(simply|just|merely) (run|execute|click|do)\\b" - why: minimizes a step that has a destructive failure mode -must_hedge: [] -required: - - pattern: "(?i)(do not|must not|never|before)" - why: there is an ordering constraint whose violation logs out every user - - pattern: "(?i)(rollback|roll back|revert|restore)" - why: the notes contain a recovery path and a 3am reader needs it ---- - -## Input - -Notes from the engineer who did it last time: - -``` -you get the new key from vault, path is secret/auth/signing, field is next_key. -it's already generated, the cron makes it monthly. - -then you set it as the active key via the admin API, PUT /admin/keys/active with -the kid. THE OLD KEY HAS TO STAY IN THE VERIFY SET or every live session breaks — -there's a separate list, verify_kids, and the old kid must be in it for at least -24h because that's the token TTL. if you revoke the old kid immediately you log -out every user, which is what happened in April. - -after 24h you remove the old kid from verify_kids. - -if the PUT fails halfway you can end up with active_kid set but verify_kids not -updated. symptom is 401s on everything. fix is PUT the old kid back as active, -then retry. - -the health endpoint /admin/keys/health shows both lists, check it after every step. -``` diff --git a/eval/technical-writing/tasks/tradeoff.md b/eval/technical-writing/tasks/tradeoff.md deleted file mode 100644 index d983700..0000000 --- a/eval/technical-writing/tasks/tradeoff.md +++ /dev/null @@ -1,61 +0,0 @@ ---- -id: tradeoff -gear: 1 -brief: > - Write up this design discussion for the team so someone joining next week understands - where the thinking currently stands. -terms: - - ["gateway", "proxy", "edge"] -forbidden: - - pattern: "(?i)^#+ *(decision|we (will|have) (decided|chosen))" - why: flattens a live disagreement into a decision that was never made - - pattern: "(?i)\\bwe (will|have) (decided|chosen|selected|agreed)\\b" - why: no decision was reached in the input - - pattern: "(?i)\\b(the )?recommended (approach|option|path) is\\b" - why: invents a recommendation neither engineer made -must_hedge: - - claim: unresolved - presence: ["APIM", "ingress", "option", "position", "approach"] - markers: ["unresolved", "disagree", "open", "not decided", "undecided", - "no decision", "still", "yet to", "tension", "stuck", "nothing was decided"] - why: the discussion ended without agreement and that is the main fact - - claim: cost_unknown - presence: ["latency", "hop", "ms", "p99", "measur", "cost"] - markers: ["unknown", "unclear", "no numbers", "not measured", "unmeasured", "guess", - "estimate", "nobody", "not been", "no one", "no data"] - why: the latency cost is explicitly unmeasured -required: - - pattern: "(?i)\\b(mira|dev)\\b" - why: attributing positions to the people holding them is what makes exploration readable ---- - -## Input - -Notes from a whiteboard session. Nothing was decided. - -``` -Question: do we put the new billing API behind the existing APIM gateway, or give it -its own ingress? - -Mira's position: APIM. We already pay for it, it already does the auth handoff, and -every other service is behind it. Standing up a second ingress means a second set of -WAF rules, a second cert rotation, a second thing that breaks at 3am. She's been burned -by exactly this at a previous job — two ingresses drifted apart over a year and nobody -noticed until an audit. - -Dev's position: own ingress. APIM adds a hop we can't tune, and billing is the one -service where p99 actually shows up in a contract. He also points out APIM's policy -language is a pain to test and the billing team can't deploy a policy change without -going through the platform team, which is a two-week queue right now. - -Where they agree: the two-week platform queue is the real problem and neither option -fixes it. - -Where they got stuck: nobody has measured what the APIM hop actually costs. Dev thinks -it's 15-40ms. Mira thinks it's under 10ms. There is no number. Somebody could measure -it in an afternoon and nobody has. - -Also raised and dropped: Kong (nobody wants to operate it), and putting billing behind -APIM but with a bypass route for the one latency-sensitive endpoint. That last one got -a "huh, maybe" from both of them and then the meeting ended. -``` From 929c3daa489d85b899be5f2c7e01ce942ffab79c Mon Sep 17 00:00:00 2001 From: Scott Pfister Date: Mon, 3 Aug 2026 14:26:50 -0500 Subject: [PATCH 10/11] fix spacing for tables --- skills/precise-technical-writing/SKILL.md | 30 +++++++++++------------ 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/skills/precise-technical-writing/SKILL.md b/skills/precise-technical-writing/SKILL.md index e4742ab..9e72fae 100644 --- a/skills/precise-technical-writing/SKILL.md +++ b/skills/precise-technical-writing/SKILL.md @@ -33,27 +33,27 @@ Classify intent first, artifact second. | Intent | Gear | | --------------------------------------------------------------- | ---- | | `instruct` — tell someone what to do | 3 | -| `specify` — state requirements, contracts, invariants, criteria | 3 | +| `specify` — state requirements, contracts, invariants, criteria | 3 | | `look-up` — help someone find a fact later | 4 | | `explain` — help someone understand | 2 | -| `justify` — give rationale, tradeoffs, or risk | 2 | -| `respond` — answer a person, in review or collaboration | 2 | -| `explore` — think through unknowns or options | 1 | +| `justify` — give rationale, tradeoffs, or risk | 2 | +| `respond` — answer a person, in review or collaboration | 2 | +| `explore` — think through unknowns or options | 1 | When intent is mixed or unclear, fall back to the artifact: -| Artifact | Gear | Shift for | -| ----------------------------------------------- | ------------------ | -------------------------------------- | -| Code comment | 3 | 2 for rationale | +| Artifact | Gear | Shift for | +| ------------------------------------------------ | ------------------ | -------------------------------------- | +| Code comment | 3 | 2 for rationale | | API doc, runbook, procedure, acceptance criteria | 3 | 2 for background | -| Agent skill, project instructions | 3 | 4 for lookup tables, 2 for context | -| PR description | 2 | 3 for testing, rollout, reviewer steps | -| Review reply | 2 | 3 for exact commitments | -| Explainer, design summary, issue body | 2 | — | -| Architecture index, module summary, repo guide | 4 | 2 for short context | -| Handoff note | 4 | 3 for next steps and commands | -| ADR | 3 for the decision | 2 for rationale, 1 for discussion | -| Brainstorm, strategy, RFC discussion | 1 | — | +| Agent skill, project instructions | 3 | 4 for lookup tables, 2 for context | +| PR description | 2 | 3 for testing, rollout, reviewer steps | +| Review reply | 2 | 3 for exact commitments | +| Explainer, design summary, issue body | 2 | — | +| Architecture index, module summary, repo guide | 4 | 2 for short context | +| Handoff note | 4 | 3 for next steps and commands | +| ADR | 3 for the decision | 2 for rationale, 1 for discussion | +| Brainstorm, strategy, RFC discussion | 1 | — | ## Gear rules From 89e02c349be55d3c4f0ed8fb0391b148d272c61f Mon Sep 17 00:00:00 2001 From: Scott Pfister Date: Mon, 3 Aug 2026 14:42:28 -0500 Subject: [PATCH 11/11] remove erroneously added file --- skills/claude-usage-report/KNOWN-ISSUES.md | 30 ---------------------- 1 file changed, 30 deletions(-) delete mode 100644 skills/claude-usage-report/KNOWN-ISSUES.md diff --git a/skills/claude-usage-report/KNOWN-ISSUES.md b/skills/claude-usage-report/KNOWN-ISSUES.md deleted file mode 100644 index f0885fe..0000000 --- a/skills/claude-usage-report/KNOWN-ISSUES.md +++ /dev/null @@ -1,30 +0,0 @@ -# Known issues / backlog - -## Day-bucketing uses UTC, not local timezone — undercounts late-evening sessions - -`usage_report.py`'s BY DAY (and single-day-argument) logic buckets each message by the -UTC date of its timestamp. For a user in a negative-UTC-offset timezone (e.g. CDT, -UTC-5), any session that runs past ~7pm local time rolls into the *next* UTC calendar -date. The report then splits that one real evening session across two "day" buckets — -and if you query a single day (e.g. `usage_report.py 2026-07-20`), you only see the -pre-midnight-UTC half, silently undercounting that day's actual spend. - -**Reproduced 2026-07-21**: querying `2026-07-20` alone showed session `b8c257de` (a CE-31 -Jira/instrumentation review) at $3.27. Widening the query to `2026-07-20..2026-07-21` -showed the same session — which ran 22:09 UTC 07-20 through 02:20 UTC 07-21, i.e. -5:09pm–9:20pm CDT, entirely within local 07-20 — actually cost $9.08. Same story for -session `6cc3db71` ($0.44 vs. $1.70 full). Together that's ~$7 of same-local-day spend -that a single-day query hid. - -**Fix**: bucket by local date instead of UTC. Simplest approach — accept a -`--tz`/config-driven UTC offset (or read the system local timezone) and convert each -message timestamp to local time before taking its date for BY DAY / single-day-argument -filtering. Also consider: the account-attribution and BY SESSION timestamp columns are -UTC-labeled and unambiguous as-is, so those probably don't need to change — only the -date-bucketing/filtering logic does. - -## BY DAY table isn't ordered by date - -`BY DAY` (and the report skeleton's "By day" table) currently prints in whatever order -the day-cost dict iterates, not chronologically. Sort ascending (or descending, pick one -and document it) by date before printing/writing the table.