Skip to content

Add autonomous contract and scale evaluation - #10

Merged
kamjcx merged 2 commits into
mainfrom
codex/autonomous-evaluation-v1
Sep 2, 2026
Merged

Add autonomous contract and scale evaluation#10
kamjcx merged 2 commits into
mainfrom
codex/autonomous-evaluation-v1

Conversation

@kamjcx

@kamjcx kamjcx commented Sep 2, 2026

Copy link
Copy Markdown
Owner

Scope

Developer-only evaluation delivery. This PR does not modify src/a1_factor_engine/, retrieval, ranking, qualification, factor IDs/values, formal catalogs, or existing frozen answers.

Added

  • independent contract Oracle and 414 deterministic public-synthetic cases
  • exact boundary/subject/unit/provenance/source-priority checks
  • metamorphic, high-risk-neighbour and eight workflow state-machine attacks
  • explicit-denominator metrics, Bad Case attribution, immutable first-run JSON/Markdown/SHA evidence
  • 10k/50k scale benchmark with c10/c25/c50 latency, RSS, throughput and stability
  • CI evidence execution and public methodology/limitations

Immutable first run

The frozen evaluator commit fe4eee7 produced 314/418 complete-row passes and an explicit NO-GO. Top-1 93.05%, Recall@5 99.23%, abstention 90%, MORE_INPUT 20%, evidence metadata 93.27%, replay 100%, zero boundary/subject/unit/HTTP-500 violations, six geography/year forbidden escapes, and 7/8 workflow attacks. All 104 Bad Cases remain unchanged.

The original Manifest git_dirty timing defect is preserved and explained by an erratum; it did not alter raw results.

Performance

10k/50k stability gates pass. At 50k: 117.67s index build, 1.409s warm P95, 205.5MB peak RSS, ~0.75 req/s one-process throughput. This is descriptive synthetic evidence, not an SLA.

Local QA

  • Python 3.11/3.12/3.13: 350 passed each
  • core branch coverage: 87.06%
  • autonomous focused: 26 passed; strict mypy/Ruff/compileall pass
  • existing FactorBench V1/V2/V3, Frozen Unit, Sealed Unit v2/v4, Portfolio Challenge: pass
  • package build/archive isolation: pass
  • local Docker daemon unavailable; remote container check is authoritative

Runtime defects, if adjudicated, will be repaired only in a separate PR after this evidence PR.

@kamjcx
kamjcx merged commit 0a69dba into main Sep 2, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant