Remediation safety assessment uplift

.specify/feature.json

@@ -1,3 +1,3 @@
 {
-  "feature_directory": "specs/011-remediation-safety-rename"
+  "feature_directory": "specs/012-remediation-safety"
 }

CHANGELOG.md

@@ -7,6 +7,29 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+
+- A ruleset analyser (`analyseRuleset()`) that assigns every rule in a loaded
+  ruleset a risk level (`low`/`medium`/`high`), a confidence level
+  (`high`/`medium`/`low`), and a derived remediation safety level
+  (`safe`/`humanreview`/`unsafe`), with provenance and a human-readable
+  rationale. See
+  [automated_remediation_safety_algorithm_spec.md](specs/algorithms/automated_remediation_safety_algorithm_spec.md).
+- `--remediation-safety <level>` (CLI) and the `grade-api-remediation-safety`
+  MCP tool's `level` parameter now accept all three levels — `safe`,
+  `humanreview`, and `unsafe` — instead of only `safe`. Every returned item now
+  also carries `riskLevel`, `confidenceLevel`, `remediationSafetyLevel`, and
+  `staleFingerprintWarning`. `safe` membership is unchanged from prior
+  behavior.
+- A new CLI subcommand, `ruleset-analysis [--ruleset-path <path>] [--format
+  json|human]`, and a new MCP tool, `analyse-ruleset-safety`, expose the
+  analyser's output independent of grading any specific spec.
+- `ruleset-analysis correct --rule-id <id> --level <level>` persists a
+  human-confirmed correction for one rule, colocated with the ruleset (or as a
+  personal override when the ruleset's location isn't locally writable), and
+  reloaded automatically on future runs against the same ruleset — including by
+  teammates pointed at the same shared ruleset.
+
 ### Changed
 
 - **Breaking**: the CLI's `--quick-fixes-only` flag is renamed to

CLAUDE.md

@@ -1,5 +1,5 @@
 <!-- SPECKIT START -->
 For additional context about technologies to be used, project structure,
 shell commands, and other important information, read the current plan
-at specs/011-remediation-safety-rename/plan.md
+at specs/012-remediation-safety/plan.md
 <!-- SPECKIT END -->

CONTRIBUTING.md

@@ -68,7 +68,7 @@ tests/
 |---------|------|-------------|
 | `@dawmatt/api-grade` | `/` (root) | CLI tool (`api-grade` binary) |
 | `@dawmatt/api-grade-core` | `packages/api-grade-core/` | Standalone grading library used by all other packages |
-| `@dawmatt/api-grade-mcp` | `packages/api-grade-mcp/` | MCP server exposing six AI tools (`grade-api`, `grade-api-detailed`, `assert-api-grade`, `grade-api-quick-fixes-only`, `set-ruleset-config`, `get-ruleset-config`) |
+| `@dawmatt/api-grade-mcp` | `packages/api-grade-mcp/` | MCP server exposing seven AI tools (`grade-api`, `grade-api-detailed`, `assert-api-grade`, `grade-api-remediation-safety`, `analyse-ruleset-safety`, `set-ruleset-config`, `get-ruleset-config`) |
 | `@dawmatt/backstage-plugin-api-grade` | `packages/backstage-plugin-api-grade/` | Backstage frontend card plugin |
 | `@dawmatt/backstage-plugin-api-grade-backend` | `packages/backstage-plugin-api-grade-backend/` | Backstage backend grading plugin |
 

docs/cli/commands.md

@@ -26,7 +26,7 @@ api-grade <spec-file> [options]
 | `--token <pat>` | GitHub Personal Access Token used to authenticate a remote ruleset fetch (only consulted when `--auth-type github-pat`) |
 | `--format <type>` | Output format: `human` (default) or `json` |
 | `--top <n>` | Show only the top N diagnostics (useful for large specs) |
-| `--remediation-safety <level>` | Filter diagnostics to the given remediation safety level (currently: `safe`) |
+| `--remediation-safety <level>` | Filter diagnostics to the given remediation safety level: `safe`, `humanreview`, or `unsafe` |
 | `--verbose` | Print the full error stack when a runtime error occurs |
 | `-V, --version` | Print the version number |
 | `-h, --help` | Show usage information |
@@ -319,9 +319,10 @@ api-grade openapi.yaml --ruleset my-rules.yaml
 > `diagnosticCounts` wrapper. See [CHANGELOG.md](../../CHANGELOG.md) for the
 > old → new field mapping.
 
-When using `--format json`, the output is a JSON object with the same flat field
-names used by the MCP server's `grade-api` / `grade-api-detailed` tools — one
-parser works for both:
+When using `--format json`, the output is **pretty-printed** (two-space indented, like every
+other end-user-visible JSON document this CLI prints — no compact/minified output) and is a
+JSON object with the same flat field names used by the MCP server's `grade-api` /
+`grade-api-detailed` tools — one parser works for both:
 
 ```json
 {
@@ -351,52 +352,93 @@ parser works for both:
       "message": "\"version\" property must be string.",
       "severity": "error",
       "path": ["info", "version"],
-      "range": { "start": { "line": 3, "character": 0 }, "end": { "line": 3, "character": 5 } }
+      "range": { "start": { "line": 3, "character": 0 }, "end": { "line": 3, "character": 5 } },
+      "source": "openapi.yaml",
+      "riskLevel": "high",
+      "confidenceLevel": "low",
+      "remediationSafetyLevel": "unsafe",
+      "staleFingerprintWarning": null
     }
   ],
   "rulesetSource": "default"
 }
 ```
 
+Every diagnostic always carries `riskLevel`, `confidenceLevel`, `remediationSafetyLevel`, and
+`staleFingerprintWarning` — the same per-violation remediation-safety signals described below
+under [Remediation Safety](#remediation-safety---remediation-safety-level) — computed from the
+ruleset analyser against the spec's effective ruleset. This is **not** gated behind
+`--remediation-safety`; it is present on every `--format json` (and, as a one-line
+`safety=... risk=... confidence=...` annotation under each finding, every `--format human`)
+grading run, so a regular user can see at a glance how risky each fix is without requesting a
+separate filtered view. `--remediation-safety <level>` (below) additionally *filters* the
+diagnostics down to one level and reshapes them into `remediationItems`; absent that flag, the
+full, unfiltered diagnostic list is annotated in place instead.
+
 `truncated: true` is added only when `--top` actually drops entries from `diagnostics`.
 `rulesetPath` is added only when a custom ruleset was used.
 
 ---
 
 ## Remediation Safety (`--remediation-safety <level>`)
 
-`--remediation-safety safe` filters diagnostics down to the non-breaking,
-safely-automatable subset — the same classification used by the MCP server's
-`grade-api-remediation-safety` tool. It is a *filter*, independent of `--format`, so it
-works with either output format. Only `safe` is accepted today; any other value is
-rejected with a non-zero exit code.
+`--remediation-safety <level>` filters diagnostics down to one of three remediation-safety
+levels — the same classification used by the MCP server's `grade-api-remediation-safety`
+tool — and is computed by the ruleset analyser (see `ruleset-analysis` below). It is a
+*filter*, independent of `--format`, so it works with either output format.
+
+| Level | Meaning |
+|---|---|
+| `safe` | Non-breaking, safe to auto-apply without per-change human review |
+| `humanreview` | Typically additive/clarifying, but should be confirmed by a human before applying at scale |
+| `unsafe` | Could change request/response validation, required fields, types, or the parameter surface — requires human (or explicitly-confirmed agent) review |
+
+Any other value is rejected with `Error: --remediation-safety must be one of: safe,
+humanreview, unsafe.` and a non-zero exit code.
 
 **Machine-readable:**
 
 ```bash
-api-grade openapi.yaml --remediation-safety safe --format json
+api-grade openapi.yaml --remediation-safety humanreview --format json
 ```
 
+This output is pretty-printed the same as the regular `--format json` output above:
+
 ```json
 {
   "specPath": "openapi.yaml",
   "format": "openapi-3",
   "totalViolations": 22,
-  "quickFixCount": 3,
-  "quickFixes": [
+  "requestedLevel": "humanreview",
+  "remediationItemCount": 3,
+  "remediationItems": [
     {
-      "ruleId": "info-contact",
-      "message": "Info object must have \"contact\" object.",
+      "ruleId": "operation-operationId",
+      "message": "Operation must have \"operationId\".",
       "severity": "warn",
-      "path": ["info"],
-      "location": "info",
+      "path": ["paths", "/pets", "get"],
+      "location": "paths./pets.get",
+      "range": { "start": { "line": 11, "character": 2 }, "end": { "line": 11, "character": 5 } },
       "currentValue": null,
-      "expectedImprovement": "Add a `contact` object to the info block with name, email, or url"
+      "expectedImprovement": "Fix: Operation must have \"operationId\". Add or update `operationId` as required",
+      "riskLevel": "medium",
+      "confidenceLevel": "high",
+      "remediationSafetyLevel": "humanreview",
+      "staleFingerprintWarning": null
     }
   ]
 }
 ```
 
+Each item carries `riskLevel` (`low`/`medium`/`high`) and `confidenceLevel`
+(`high`/`medium`/`low`) alongside `remediationSafetyLevel` — the field
+`--remediation-safety`/`requestedLevel` filters against — and a `staleFingerprintWarning`
+that is non-null only when a human-assessed rule classification's underlying rule
+definition has since changed (see `ruleset-analysis` below). `severity` is the diagnostic's
+actual severity (`error`/`warn`/`info`/`hint`, not a fixed placeholder), and `range` carries
+the same line/character location as the regular diagnostics output — both are required to
+act on a remediation item without losing the line-number context a linter normally provides.
+
 **Human-readable** (default, or with `--format human`):
 
 ```bash
@@ -405,11 +447,42 @@ api-grade openapi.yaml --remediation-safety safe
 
 Prints the same filtered list as readable text instead of JSON.
 
-`--remediation-safety safe` has no effect on `--min-grade` — the gate still evaluates the
+`--remediation-safety <level>` has no effect on `--min-grade` — the gate still evaluates the
 spec's actual letter grade from the full, unfiltered diagnostics.
 
 ---
 
+## Ruleset Analysis (`ruleset-analysis`)
+
+Inspects a ruleset's per-rule remediation-safety analysis independent of grading any spec:
+
+```bash
+api-grade ruleset-analysis --format human
+api-grade ruleset-analysis --ruleset-path ./my-ruleset.yaml --format json
+```
+
+`--format human` (default) prints a table with rule id, risk level, confidence level,
+remediation safety level, assessed by (`human`/`automated`), and rationale — plus a
+fingerprint-mismatch warning line for any human-assessed rule whose underlying definition
+has changed since it was last reviewed. `--format json` returns the full `RulesetAnalysis`
+document. Without `--ruleset-path`, analyses the built-in ruleset.
+
+To persist a human-confirmed correction for one rule (reloaded automatically on future runs
+against the same ruleset):
+
+```bash
+api-grade ruleset-analysis correct --rule-id operation-operationId --level safe \
+  --ruleset-path ./my-ruleset.yaml
+```
+
+For a local ruleset, this writes a colocated `<ruleset>.remediation-safety.json` file next
+to the ruleset (commit it so your team shares the same judgements). For a non-writable
+ruleset location (e.g. a GitHub-hosted ruleset, or the built-in ruleset), the correction is
+recorded locally as a personal override instead, and the equivalent shared-file content is
+printed for you to commit yourself.
+
+---
+
 ## Structured `--min-grade` Outcome in JSON Mode
 
 When `--min-grade <LETTER>` is combined with `--format json`, the CLI prints a

docs/getting-started.md

@@ -55,7 +55,7 @@ Two Backstage plugin packages that display API grades directly on your Backstage
 
 ### MCP Server (`@dawmatt/api-grade-mcp`)
 
-An MCP (Model Context Protocol) server that exposes api-grade as six AI tools: `grade-api`, `grade-api-detailed`, `assert-api-grade`, `grade-api-remediation-safety`, `set-ruleset-config`, and `get-ruleset-config`. Register it in Claude Code, GitHub Copilot (VS Code Agent mode), or any MCP-compatible AI host and let the AI grade specs directly.
+An MCP (Model Context Protocol) server that exposes api-grade as seven AI tools: `grade-api`, `grade-api-detailed`, `assert-api-grade`, `grade-api-remediation-safety`, `analyse-ruleset-safety`, `set-ruleset-config`, and `get-ruleset-config`. Register it in Claude Code, GitHub Copilot (VS Code Agent mode), or any MCP-compatible AI host and let the AI grade specs directly.
 
 ```bash
 claude mcp add api-grade -- npx -y @dawmatt/api-grade-mcp

docs/index.md

@@ -15,9 +15,9 @@
 | [Package Usage Guide](package/usage-guide.md) | Common integration patterns and worked examples |
 | [Package API Reference](package/api-reference.md) | All exported functions, classes, and types |
 | [API Diagnostic Algorithm Specification](../specs/algorithms/api_diagnostic_algorithm_spec.md) | How scores, grades, and recommendations are computed |
-| [Quick-Fixes Algorithm Specification](../specs/algorithms/quick_fixes_algorithm_spec.md) | How non-breaking, safely-automatable violations are identified |
+| [Automated Remediation Safety Algorithm Specification](../specs/algorithms/automated_remediation_safety_algorithm_spec.md) | How risk, confidence, and remediation safety are determined per rule |
 | [MCP Server](mcp/README.md) | Grade specs from AI tools via MCP |
-| [MCP Server Overview](package/api-grade-mcp.md) | All six MCP tools and their inputs/outputs |
+| [MCP Server Overview](package/api-grade-mcp.md) | All MCP tools and their inputs/outputs |
 | [MCP Quick Start](mcp/quick-start.md) | Install and configure the MCP server in minutes |
 | [MCP Configuration Reference](mcp/configuration.md) | Default rulesets, auth, and scope precedence |
 | [MCP GitHub Token Setup](mcp/github-pat-setup.md) | One-time GitHub PAT creation for `github-pat` ruleset auth |

docs/mcp/quick-start.md

@@ -175,7 +175,8 @@ Reload Cursor after saving.
 | `grade-api` | Quick grade: letter grade, numeric score, and summary |
 | `grade-api-detailed` | Full grade with all violations, diagnostics, and recommendations |
 | `assert-api-grade` | Pass/fail assertion for a minimum grade threshold |
-| `grade-api-remediation-safety` | Classified list of diagnostics filtered by remediation safety level (`safe`: non-breaking improvements) for AI-assisted correction |
+| `grade-api-remediation-safety` | Classified list of diagnostics filtered by remediation safety level (`safe`, `humanreview`, or `unsafe`), each with a risk/confidence indicator, for AI-assisted correction |
+| `analyse-ruleset-safety` | Per-rule risk, confidence, and remediation-safety analysis for a ruleset, independent of grading any spec |
 | `set-ruleset-config` | Set the default Spectral ruleset at session, workspace, or global scope |
 | `get-ruleset-config` | Get the active Spectral ruleset and which scope is effective |
 
@@ -207,7 +208,7 @@ To confirm the server starts correctly:
 echo '{"jsonrpc":"2.0","id":1,"method":"tools/list","params":{}}' | npx -y @dawmatt/api-grade-mcp
 ```
 
-You should see a JSON response listing all six tools.
+You should see a JSON response listing all the tools above.
 
 ---
 

docs/package/README.md

@@ -79,6 +79,6 @@ After building, the `api-grade-core` package is available under `packages/api-gr
 - [Usage Guide](usage-guide.md) — common integration patterns and worked examples
 - [API Reference](api-reference.md) — all exported functions, classes, and types
 - [API Diagnostic Algorithm Specification](../../specs/algorithms/api_diagnostic_algorithm_spec.md) — how scores, grades, and recommendations are computed
-- [Quick-Fixes Algorithm Specification](../../specs/algorithms/quick_fixes_algorithm_spec.md) — how non-breaking, safely-automatable violations are identified
+- [Automated Remediation Safety Algorithm Specification](../../specs/algorithms/automated_remediation_safety_algorithm_spec.md) — how risk, confidence, and remediation safety are determined per rule
 - [Documentation Index](../index.md) — full navigation across all docs
 - [CLI Tool](../cli/README.md) — use api-grade from the command line

docs/package/api-grade-mcp.md

@@ -121,14 +121,24 @@ Assert that an API specification meets a minimum grade threshold (A > B > C > D
 
 ### `grade-api-remediation-safety`
 
-Return a classified, AI-actionable list of diagnostics filtered by remediation safety level. The `safe` level covers improvements that can be made via non-breaking changes (those that do not alter paths, methods, required parameters, schema types, or response structures). Each result includes `ruleId`, `path`, `location`, `currentValue`, and `expectedImprovement`.
+Return a classified, AI-actionable list of diagnostics filtered by remediation safety level: `safe` (non-breaking, safe to auto-apply), `humanreview` (typically additive/clarifying but should be confirmed by a human before applying at scale), or `unsafe` (could change request/response validation, required fields, types, or the parameter surface — requires human or explicitly-confirmed-agent review). Each result includes `ruleId`, `severity`, `path`, `location`, `range`, `currentValue`, `expectedImprovement`, and a confidence indicator (`riskLevel`, `confidenceLevel`, `remediationSafetyLevel`, `staleFingerprintWarning`) — `severity` and `range` are carried over unchanged from the underlying diagnostic, so no line-number/severity context is lost relative to `grade-api-detailed`.
 
-**Input**: `specPath` (required), `level` (required: `safe`), `rulesetPath` (optional), `recoveryOption` (optional)
+**Input**: `specPath` (required), `level` (required: `safe`/`humanreview`/`unsafe`), `rulesetPath` (optional), `recoveryOption` (optional)
 
 **Use when**: Asking the AI to generate fixes for documentation and metadata issues without risking breaking changes. Use this tool instead of `grade-api-detailed` when the goal is AI-assisted correction.
 
 ---
 
+### `analyse-ruleset-safety`
+
+Inspect a Spectral ruleset's per-rule remediation-safety analysis (`riskLevel`, `confidenceLevel`, `remediationSafetyLevel`, `assessedBy`, `rationale`) without grading any specific API specification. Returns the same `RulesetAnalysis` document the CLI's `ruleset-analysis` subcommand produces.
+
+**Input**: `rulesetPath` (optional), `recoveryOption` (optional)
+
+**Use when**: You want to understand how risky it would be to auto-remediate violations of each rule in a ruleset before running `grade-api-remediation-safety` against a real spec.
+
+---
+
 ### `set-ruleset-config`
 
 Set the default Spectral ruleset at session, workspace, or global scope. The configured default applies to all subsequent grading requests without needing to supply `rulesetPath` each time.