quality(routing): earn the representative-corpus gate

README.md

@@ -6,7 +6,7 @@
 ./script/build_and_run.sh
 ```
 
-[See exactly how the product extends Apple’s WWDC26 file-sorting demo, including the reproducible negative benchmark results.](docs/wwdc26-comparison.md)
+[See exactly how the product extends Apple’s WWDC26 file-sorting demo, including the passing shipping-path benchmark and its limitations.](docs/wwdc26-comparison.md)
 
 > [!WARNING]
 > The current `v0.1.0` build is an experimental pre-release. Its app bundle is ad-hoc signed, not signed with an Apple Developer ID, and it is not notarized. Gatekeeper may block the first launch; managed Macs may prohibit it entirely. Homebrew installation confirms the archive and cask are valid, but does not bypass these macOS security checks.
@@ -69,7 +69,7 @@ Sorting Hat reads searchable PDFs, plain-text formats, RTF, Word, and OpenDocume
 
 When Apple `fm` is selected, compatible non-image files are analyzed in batches of at most 8 files and 24,000 extracted characters. Each file is identified by an opaque request-local ID, and every returned decision is independently validated before any move. Missing, duplicate, unexpected, or unsafe decisions cannot be applied. Images remain on the individual multimodal path, and a failed batch does not prevent files in other batches from being processed. A deterministic process benchmark verifies that 8 compatible files use 1 `fm respond` invocation instead of 8.
 
-The Inbox is intake-only. Sorting Hat renames each file and moves it to a rule-specific folder under `output` (for example, `~/SortingHat/Receipts/2026`). It creates those destination folders as needed, rejects absolute paths and traversal, preserves existing files with numbered names, and writes tags as Finder metadata.
+The Inbox is intake-only. Sorting Hat renames each file and moves it to a rule-specific folder under `output` (for example, `~/SortingHat/Receipts/2026`). For the rule builder's controlled `Put ... in ...` routes, deterministic Swift code canonicalises the model's proposal against configured destinations and rejects unknown or unresolved folders. When at least one controlled route is present, its compiled destinations form the authoritative allow-list for that ruleset; other prose can still guide naming, tagging, and classification but cannot introduce a new destination. Sorting Hat creates destination folders as needed, rejects absolute paths and traversal, preserves the source file type, protects existing files with numbered names, and writes tags as Finder metadata.
 
 If a model returns the uploaded filename unchanged, Sorting Hat leaves the source in the Inbox and reports an invalid sorting decision instead of silently skipping the requested rename.
 
@@ -97,9 +97,11 @@ Live quality checks are deliberately opt-in and read-only. Create a private, ano
 
 The manifest is versioned and contains rules plus cases with a relative source path, content kind (`receipt`, `scan`, `screenshot`, `pdf`, `office_document`, or `ambiguous`), accepted folders, required filename terms, required tags, and whether an empty-folder abstention is expected. The checked-in template defines all six required classes but deliberately includes no documents. Corpus files must sit beside or below the manifest; output inside that corpus directory is rejected. The evaluator calls Apple `fm` but never calls `Organizer.apply`, so sources are not renamed, tagged, or moved.
 
-Each run writes `evaluation.json` and `summary.md`. The artifact records model, OS, prompt version, use case, guardrails, per-case decisions and latency, accuracy, generation failures, unsafe/invalid decisions, and abstentions. Pass a previous `evaluation.json` with `--baseline` to expose regressions in the Markdown summary. Exit status `2` means a threshold or baseline regressed; `1` means the evaluation could not run. Do not commit corpus documents or results containing private or copyrighted content; only the synthetic manifest example belongs in the repository.
+Each run writes `evaluation.json` and `summary.md`. In schema-v2 artifacts, `raw_decision` is the model's direct proposal before deterministic routing policy, while `decision` is the resolved and validated shipping-path result that the evaluator scores. The artifact also records model, OS, prompt and routing-policy versions, validation errors, pre-validation decision latency, accuracy, generation failures, unsafe/invalid decisions, and abstentions. Latency stops after analysis and deterministic routing resolution, before the separate `Organizer` validation pass. Pass a schema-compatible previous `evaluation.json` with `--baseline` to expose regressions; incompatible schemas, corpora, prompts, routing policies, or model environments are reported instead of being compared silently. Exit status `2` means a threshold or baseline check failed; `1` means the evaluation could not run. Do not commit corpus documents or results containing private or copyrighted content; only the synthetic manifest example belongs in the repository.
 
-For reproducible multi-prompt, system/content-tagging, and system/PCC comparisons, use the locked standalone Python project in [`evaluation/`](evaluation/README.md). It consumes the same corpus manifest and produces JSON, CSV, Markdown, and chart artifacts without adding Python to the application runtime.
+The completed Issue #23 result is published in [`evaluation/ROUTING_RESULTS.md`](evaluation/ROUTING_RESULTS.md): routing policy v1 scored 108/108 exact decisions across nine final runs, with 18/18 ambiguous cases held for review, zero invalid final decisions, and an 8.1% recorded pre-validation latency increase against the corrected shipping-path baseline. The 12-case private corpus is a regression gate, not a universal accuracy claim.
+
+For reproducible prompt, system/content-tagging, and system/PCC research, use the locked standalone Python project in [`evaluation/`](evaluation/README.md). It consumes the same corpus manifest and produces JSON, CSV, Markdown, and chart artifacts without adding Python to the application runtime. The Swift live evaluator remains the product-quality authority because it exercises the shipping extractor, routing policy, and validator.
 
 Bounded tool-calling candidates are also evaluated there under a documented threat model and evidence gate. Filesystem mutation remains exclusively in validated Swift code; no evaluation tool is available to the shipping app unless a future change demonstrates and reviews measurable value first.
 

Sources/SortingHat/CLI.swift

@@ -56,7 +56,8 @@ enum SortingHatCLI {
                                   guardrails: config.appleGuardrails, pccAllowed: config.allowApplePCC)
         let configuration = EvaluationConfiguration(model: model.rawValue, useCase: config.appleUseCase.rawValue,
             guardrails: config.appleGuardrails.rawValue, pccAllowed: config.allowApplePCC,
-            promptVersion: FMAnalyzer.promptVersion, operatingSystem: ProcessInfo.processInfo.operatingSystemVersionString)
+            promptVersion: FMAnalyzer.promptVersion, operatingSystem: ProcessInfo.processInfo.operatingSystemVersionString,
+            routingPolicyVersion: RoutingDecisionResolver.version)
         let baseline: EvaluationArtifact?
         if let path = value(after: "--baseline", in: args) {
             let decoder = JSONDecoder(); decoder.dateDecodingStrategy = .iso8601

Sources/SortingHatCore/LiveEvaluation.swift

@@ -46,14 +46,24 @@ public struct EvaluationConfiguration: Codable, Sendable {
     public let pccAllowed: Bool
     public let promptVersion: String
     public let operatingSystem: String
+    public let routingPolicyVersion: String?
 
-    public init(model: String, useCase: String, guardrails: String, pccAllowed: Bool, promptVersion: String, operatingSystem: String) {
+    public init(
+        model: String,
+        useCase: String,
+        guardrails: String,
+        pccAllowed: Bool,
+        promptVersion: String,
+        operatingSystem: String,
+        routingPolicyVersion: String? = nil
+    ) {
         self.model = model
         self.useCase = useCase
         self.guardrails = guardrails
         self.pccAllowed = pccAllowed
         self.promptVersion = promptVersion
         self.operatingSystem = operatingSystem
+        self.routingPolicyVersion = routingPolicyVersion
     }
 
     enum CodingKeys: String, CodingKey {
@@ -62,6 +72,7 @@ public struct EvaluationConfiguration: Codable, Sendable {
         case pccAllowed = "pcc_allowed"
         case promptVersion = "prompt_version"
         case operatingSystem = "operating_system"
+        case routingPolicyVersion = "routing_policy_version"
     }
 }
 
@@ -98,6 +109,7 @@ public struct EvaluationResult: Codable, Sendable {
     public let id: String
     public let kind: String
     public let latencyMilliseconds: Double
+    public let rawDecision: Decision?
     public let decision: Decision?
     public let error: String?
     public let folderCorrect: Bool
@@ -108,6 +120,7 @@ public struct EvaluationResult: Codable, Sendable {
 
     enum CodingKeys: String, CodingKey {
         case id, kind, decision, error, abstained
+        case rawDecision = "raw_decision"
         case latencyMilliseconds = "latency_ms"
         case folderCorrect = "folder_correct"
         case filenameCorrect = "filename_correct"
@@ -160,34 +173,58 @@ public enum LiveEvaluator {
         baseline: EvaluationArtifact? = nil
     ) -> EvaluationArtifact {
         let root = corpusRoot.standardizedFileURL
+        let referenceDate = Date()
+        let recordedConfiguration = EvaluationConfiguration(
+            model: configuration.model,
+            useCase: configuration.useCase,
+            guardrails: configuration.guardrails,
+            pccAllowed: configuration.pccAllowed,
+            promptVersion: configuration.promptVersion,
+            operatingSystem: configuration.operatingSystem,
+            routingPolicyVersion: RoutingDecisionResolver.version
+        )
         let results = manifest.cases.map { item -> EvaluationResult in
             let file = root.appending(path: item.path).standardizedFileURL
             let started = ContinuousClock.now
+            var rawDecision: Decision?
             do {
-                let decision = try analyzer.analyze(file: file, rules: manifest.rules)
+                let raw = try analyzer.analyze(file: file, rules: manifest.rules)
+                rawDecision = raw
+                let decision = try RoutingDecisionResolver.resolve(
+                    file: file,
+                    decision: raw,
+                    rules: manifest.rules,
+                    referenceDate: referenceDate
+                )
                 let elapsed = milliseconds(since: started)
                 let abstained = decision.folder.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty
-                let folderCorrect = item.expected.abstain ? abstained : item.expected.folders.contains(decision.folder)
+                let validationError = validate(
+                    file: file,
+                    decision: decision,
+                    rules: manifest.rules,
+                    referenceDate: referenceDate
+                )
+                let valid = validationError == nil
+                let folderCorrect = valid && (item.expected.abstain ? abstained : item.expected.folders.contains(decision.folder))
                 let loweredName = decision.filename.lowercased()
-                let filenameCorrect = item.expected.filenameContains.allSatisfy { loweredName.contains($0.lowercased()) }
+                let filenameCorrect = valid && item.expected.filenameContains.allSatisfy { loweredName.contains($0.lowercased()) }
                 let loweredTags = Set(decision.tags.map { $0.lowercased() })
-                let tagsCorrect = item.expected.tags.allSatisfy { loweredTags.contains($0.lowercased()) }
-                let invalid = validate(file: file, decision: decision, rules: manifest.rules) != nil
-                return EvaluationResult(id: item.id, kind: item.kind, latencyMilliseconds: elapsed, decision: decision,
-                                        error: nil, folderCorrect: folderCorrect, filenameCorrect: filenameCorrect,
-                                        tagsCorrect: tagsCorrect, abstained: abstained, unsafeOrInvalid: invalid)
+                let tagsCorrect = valid && item.expected.tags.allSatisfy { loweredTags.contains($0.lowercased()) }
+                return EvaluationResult(id: item.id, kind: item.kind, latencyMilliseconds: elapsed, rawDecision: rawDecision, decision: decision,
+                                        error: validationError?.localizedDescription, folderCorrect: folderCorrect, filenameCorrect: filenameCorrect,
+                                        tagsCorrect: tagsCorrect, abstained: abstained, unsafeOrInvalid: !valid)
             } catch {
                 return EvaluationResult(id: item.id, kind: item.kind, latencyMilliseconds: milliseconds(since: started),
-                                        decision: nil, error: error.localizedDescription, folderCorrect: false,
+                                        rawDecision: rawDecision, decision: nil, error: error.localizedDescription, folderCorrect: false,
                                         filenameCorrect: false, tagsCorrect: false, abstained: false,
                                         unsafeOrInvalid: isUnsafeOrInvalid(error))
             }
         }
         let metrics = metrics(for: results)
-        return EvaluationArtifact(schemaVersion: 1, corpusName: manifest.name, createdAt: Date(), configuration: configuration,
+        return EvaluationArtifact(schemaVersion: 2, corpusName: manifest.name, createdAt: referenceDate, configuration: recordedConfiguration,
                                   metrics: metrics, results: results,
                                   thresholdFailures: thresholdFailures(metrics, manifest.thresholds),
-                                  regressions: regressions(metrics, baseline?.metrics))
+                                  regressions: regressions(metrics, baseline: baseline, corpusName: manifest.name, configuration: recordedConfiguration))
     }
 
     public static func write(_ artifact: EvaluationArtifact, to outputDirectory: URL) throws {
@@ -208,6 +245,7 @@ public enum LiveEvaluator {
         Corpus: \(artifact.corpusName)
         Model: \(artifact.configuration.model) (\(artifact.configuration.useCase))
         Prompt: \(artifact.configuration.promptVersion)
+        Routing policy: \(artifact.configuration.routingPolicyVersion ?? "legacy")
         OS: \(artifact.configuration.operatingSystem)
 
         | Metric | Result |
@@ -220,21 +258,29 @@ public enum LiveEvaluator {
         | Schema failures | \(artifact.metrics.schemaFailures) |
         | Unsafe/invalid decisions | \(artifact.metrics.unsafeOrInvalidDecisions) |
         | Abstentions | \(artifact.metrics.abstentions) |
-        | Average latency | \(String(format: "%.1f ms", artifact.metrics.averageLatencyMilliseconds)) |
+        | Average pre-validation latency | \(String(format: "%.1f ms", artifact.metrics.averageLatencyMilliseconds)) |
 
         ## Thresholds and regressions
 
         \(issues.isEmpty ? "None." : issues)
         """
     }
 
-    private static func validate(file: URL, decision: Decision, rules: [String]) -> Error? {
+    private static func validate(file: URL, decision: Decision, rules: [String], referenceDate: Date) -> Error? {
         if decision.folder.trimmingCharacters(in: .whitespacesAndNewlines).isEmpty { return nil }
         struct FixedAnalyzer: FileAnalyzing {
             let decision: Decision
             func analyze(file: URL, rules: [String]) throws -> Decision { decision }
         }
-        do { _ = try Organizer(inbox: file.deletingLastPathComponent(), rules: rules, analyzer: FixedAnalyzer(decision: decision)).plan(file); return nil }
+        do {
+            _ = try Organizer(
+                inbox: file.deletingLastPathComponent(),
+                rules: rules,
+                analyzer: FixedAnalyzer(decision: decision),
+                referenceDate: referenceDate
+            ).plan(file)
+            return nil
+        }
         catch { return error }
     }
 
@@ -262,12 +308,35 @@ public enum LiveEvaluator {
         return failures
     }
 
-    private static func regressions(_ metrics: EvaluationMetrics, _ baseline: EvaluationMetrics?) -> [String] {
+    private static func regressions(
+        _ metrics: EvaluationMetrics,
+        baseline: EvaluationArtifact?,
+        corpusName: String,
+        configuration: EvaluationConfiguration
+    ) -> [String] {
         guard let baseline else { return [] }
+        guard baseline.schemaVersion == 2 else {
+            return ["baseline artifact schema \(baseline.schemaVersion) is not comparable with schema 2"]
+        }
+        guard baseline.corpusName == corpusName, baseline.metrics.total == metrics.total else {
+            return ["baseline corpus does not match this evaluation"]
+        }
+        let baselineConfiguration = baseline.configuration
+        guard baselineConfiguration.model == configuration.model,
+              baselineConfiguration.useCase == configuration.useCase,
+              baselineConfiguration.guardrails == configuration.guardrails,
+              baselineConfiguration.pccAllowed == configuration.pccAllowed,
+              baselineConfiguration.operatingSystem == configuration.operatingSystem,
+              baselineConfiguration.promptVersion == configuration.promptVersion,
+              baselineConfiguration.routingPolicyVersion == configuration.routingPolicyVersion else {
+            return ["baseline evaluation configuration does not match this evaluation"]
+        }
+
         var values: [String] = []
-        if metrics.accuracy < baseline.accuracy { values.append("accuracy regressed from \(percent(baseline.accuracy)) to \(percent(metrics.accuracy))") }
-        if metrics.generationFailureRate > baseline.generationFailureRate { values.append("generation failure rate regressed") }
-        if metrics.unsafeDecisionRate > baseline.unsafeDecisionRate { values.append("unsafe/invalid decision rate regressed") }
+        let baselineMetrics = baseline.metrics
+        if metrics.accuracy < baselineMetrics.accuracy { values.append("accuracy regressed from \(percent(baselineMetrics.accuracy)) to \(percent(metrics.accuracy))") }
+        if metrics.generationFailureRate > baselineMetrics.generationFailureRate { values.append("generation failure rate regressed") }
+        if metrics.unsafeDecisionRate > baselineMetrics.unsafeDecisionRate { values.append("unsafe/invalid decision rate regressed") }
         return values
     }