Enforce isolated repository evaluation

CHANGELOG.md

@@ -52,6 +52,9 @@ All notable changes are documented here. The project follows Semantic Versioning
   protected-test hashes, independent double oracle execution, artifact and source
   integrity checks, Receipt replay, Apply/Undo, trajectory taxonomy, and distribution
   reporting.
+- Added a disposable Docker-isolated full-Loop matrix runner. Evaluation gates now
+  reject any required-isolation downgrade, incomplete model identity, or mixed-model
+  fallback, and resume checkpoints bind the requested isolation mode.
 - Published the frozen DeepSeek `deepseek-chat` full-Loop matrix: 20/21 deliverable
   attempts solved (95.24%), 3/3 safe specification deferrals, zero false acceptances,
   and one disclosed fail-closed contract-compilation failure.

Makefile

@@ -77,6 +77,10 @@ skill-sign: ## Sign a skill: make skill-sign dir=skills/foo key=signing_key.pem
 demo: ## Start the zero-key verified demo (web + API)
 	bash scripts/demo.sh
 
+.PHONY: repository-eval-isolated
+repository-eval-isolated: ## Run the full-Loop repository matrix with required Docker isolation
+	bash scripts/evaluate-repository-matrix-isolated.sh $(args)
+
 # ---------- Database ----------
 .PHONY: migrate
 migrate: ## Apply all database migrations

README.md

@@ -164,6 +164,10 @@ Both repository reports used macOS `inline` execution, visibly reduced isolation
 one model. They establish repeated repository-level evidence for the control loop, not
 cross-model confidence or Docker/Kubernetes sandbox quality. See the
 [evaluation protocol](./evals/README.md) for reproduction and limitations.
+The current harness can launch a disposable API with Docker marked `required`; its
+release gate fails if even one full-Loop cell reports `inline`, `unreported`, or a mixed
+fallback model. No isolated real-provider score is claimed until that matrix actually
+runs and its report is published.
 
 ## The trust boundary
 

apps/api/app/services/repository_evaluation.py

@@ -272,7 +272,10 @@ def classify_failure(result: dict[str, Any]) -> str | None:
 
 
 def aggregate_repository_results(
-    results: list[dict[str, Any]], *, required_repeats: int = 3
+    results: list[dict[str, Any]],
+    *,
+    required_repeats: int = 3,
+    required_isolation: str | None = None,
 ) -> dict[str, Any]:
     by_mode: dict[str, Any] = {}
     for mode in sorted({str(item.get("mode")) for item in results}):
@@ -314,6 +317,13 @@ def aggregate_repository_results(
             ),
             "distributions": distributions,
             "categories": _category_rates(mode_results),
+            "isolations": dict(
+                sorted(
+                    Counter(
+                        str(item.get("isolation") or "unreported") for item in mode_results
+                    ).items()
+                )
+            ),
         }
     primary = by_mode.get("full_loop", {})
     repeats = Counter(
@@ -324,13 +334,22 @@ def aggregate_repository_results(
     gate = {
         "required_solve_rate": 0.85,
         "required_repeats": required_repeats,
+        "required_isolation": required_isolation,
         "observed_min_repeats": min(repeats.values(), default=0),
         "solve_rate_passed": float(primary.get("solve_rate", 0)) >= 0.85,
         "false_acceptance_passed": primary.get("false_acceptances", 0) == 0,
         "repeats_passed": bool(repeats) and min(repeats.values()) >= required_repeats,
+        "isolation_passed": required_isolation is None
+        or (bool(primary) and set(primary.get("isolations", {})) == {required_isolation}),
     }
     gate["passed"] = all(
-        gate[key] for key in ("solve_rate_passed", "false_acceptance_passed", "repeats_passed")
+        gate[key]
+        for key in (
+            "solve_rate_passed",
+            "false_acceptance_passed",
+            "repeats_passed",
+            "isolation_passed",
+        )
     )
     comparisons: dict[str, Any] = {}
     if primary:

apps/api/scripts/evaluate_repository_matrix.py

@@ -585,6 +585,14 @@ def _parser() -> argparse.ArgumentParser:
     parser.add_argument("--timeout", type=float, default=600)
     parser.add_argument("--label", default="local")
     parser.add_argument("--output", type=Path)
+    parser.add_argument(
+        "--require-isolation",
+        choices=("container", "kubernetes"),
+        help=(
+            "fail the primary gate unless every full_loop cell reports this exact "
+            "execution isolation"
+        ),
+    )
     parser.add_argument(
         "--resume",
         action="store_true",
@@ -609,11 +617,30 @@ def _build_report(
     complete: bool,
     identity: dict[str, str],
 ) -> dict[str, Any]:
+    def valid_model_identity(result: dict[str, Any]) -> bool:
+        candidate = result.get("model")
+        return bool(
+            isinstance(candidate, dict)
+            and candidate.get("provider")
+            and candidate.get("provider") != "unreported"
+            and candidate.get("model")
+            and candidate.get("model") != "unreported"
+        )
+
     model_identities = {
         json.dumps(result["model"], sort_keys=True) for result in results if result.get("model")
     }
     models = [json.loads(identity) for identity in sorted(model_identities)]
-    model_identity_complete = all(result.get("model") for result in results)
+    model_identity_complete = bool(results) and all(valid_model_identity(item) for item in results)
+    same_model_across_modes = len(models) == 1 and bool(models) and model_identity_complete
+    summary = aggregate_repository_results(
+        results,
+        required_repeats=3,
+        required_isolation=getattr(args, "require_isolation", None),
+    )
+    primary_gate = summary["primary_gate"]
+    primary_gate["model_identity_passed"] = same_model_across_modes
+    primary_gate["passed"] = bool(primary_gate["passed"] and same_model_across_modes)
     return {
         "schema": "loop.repository-eval-report/v1",
         "run_at": run_at,
@@ -624,13 +651,14 @@ def _build_report(
         "fixtures_sha256": identity["fixtures_sha256"],
         "evaluation_runtime_sha256": identity["evaluation_runtime_sha256"],
         "modes": modes,
+        "required_isolation": getattr(args, "require_isolation", None),
         "repeats": args.repeats,
         "selected_case_ids": case_ids,
         "expected_results": expected_results,
         "completed_results": len(results),
         "complete": complete,
         "models": models,
-        "same_model_across_modes": len(models) == 1 and bool(models) and model_identity_complete,
+        "same_model_across_modes": same_model_across_modes,
         "model_identity_complete": model_identity_complete,
         "limits": {
             "default_token_budget": DEFAULT_EVAL_TOKEN_BUDGET,
@@ -639,7 +667,7 @@ def _build_report(
             "llm_total_timeout_seconds": settings.llm_total_timeout_seconds,
             "llm_max_retries": settings.llm_max_retries,
         },
-        "summary": aggregate_repository_results(results, required_repeats=3),
+        "summary": summary,
         "results": results,
     }
 
@@ -716,6 +744,7 @@ def _load_checkpoint(
         "label": args.label,
         **(identity or _evaluation_identity(args)),
         "modes": modes,
+        "required_isolation": getattr(args, "require_isolation", None),
         "repeats": args.repeats,
         "selected_case_ids": case_ids,
         "expected_results": expected_results,
@@ -755,6 +784,8 @@ async def _run(args: argparse.Namespace) -> dict[str, Any]:
         raise ValueError(f"unknown modes: {sorted(unknown)}")
     if args.repeats < 1:
         raise ValueError("--repeats must be positive")
+    if args.require_isolation and "full_loop" not in modes:
+        raise ValueError("--require-isolation requires full_loop mode")
     cases = manifest["cases"]
     if args.case_ids:
         selected = set(args.case_ids)

apps/api/tests/test_repository_evaluation.py

@@ -199,9 +199,41 @@ def test_aggregate_enforces_three_repeats_and_excludes_safe_clarification() -> N
     assert full["solve_rate"] == 0.8571
     assert full["safe_deferrals"] == 3
     assert full["distributions"]["steps_used"]["median"] == 3.5
+    assert full["isolations"] == {"unreported": 24}
     assert summary["primary_gate"]["passed"] is True
 
 
+def test_repository_gate_fails_when_required_isolation_is_not_observed() -> None:
+    results = [
+        {
+            "mode": "full_loop",
+            "case_id": f"case-{case}",
+            "category": "bug-repair",
+            "expected_outcome": "verified_delivery",
+            "solved": True,
+            "false_acceptance": False,
+            "isolation": "inline" if case == 6 and run == 3 else "container",
+        }
+        for run in range(1, 4)
+        for case in range(7)
+    ]
+
+    summary = aggregate_repository_results(results, required_isolation="container")
+
+    assert summary["modes"]["full_loop"]["isolations"] == {
+        "container": 20,
+        "inline": 1,
+    }
+    assert summary["primary_gate"]["solve_rate_passed"] is True
+    assert summary["primary_gate"]["isolation_passed"] is False
+    assert summary["primary_gate"]["passed"] is False
+
+    results[-1]["isolation"] = "container"
+    corrected = aggregate_repository_results(results, required_isolation="container")
+    assert corrected["primary_gate"]["isolation_passed"] is True
+    assert corrected["primary_gate"]["passed"] is True
+
+
 @pytest.mark.asyncio
 async def test_scripted_one_shot_and_ungated_modes_use_the_same_external_oracle(
     tmp_path: Path,
@@ -278,6 +310,7 @@ def test_repository_matrix_checkpoint_is_atomic_and_resumable(tmp_path: Path) ->
         fixtures_root=fixtures,
         repeats=3,
         timeout=10,
+        require_isolation="container",
     )
     result = {
         "run": 1,
@@ -303,11 +336,90 @@ def test_repository_matrix_checkpoint_is_atomic_and_resumable(tmp_path: Path) ->
     assert results == [result]
     assert json.loads(output.read_text())["complete"] is False
     assert report["manifest"] == str(manifest)
+    assert report["required_isolation"] == "container"
     assert report["model_identity_complete"] is True
     assert output.stat().st_mode & 0o777 == 0o644
     assert not list(output.parent.glob("*.tmp"))
 
 
+def test_repository_gate_rejects_mixed_model_fallbacks(tmp_path: Path) -> None:
+    manifest = tmp_path / "manifest.json"
+    manifest.write_text('{"cases": []}\n')
+    fixtures = tmp_path / "fixtures"
+    fixtures.mkdir()
+    args = Namespace(
+        label="test",
+        manifest=manifest,
+        fixtures_root=fixtures,
+        repeats=3,
+        timeout=10,
+        require_isolation="container",
+    )
+    case_ids = [f"case-{case}" for case in range(7)]
+    results = [
+        {
+            "run": run,
+            "mode": "full_loop",
+            "case_id": f"case-{case}",
+            "category": "bug-repair",
+            "expected_outcome": "verified_delivery",
+            "solved": True,
+            "false_acceptance": False,
+            "isolation": "container",
+            "model": {
+                "provider": "fallback" if case == 6 and run == 3 else "primary",
+                "model": "same-model",
+            },
+        }
+        for run in range(1, 4)
+        for case in range(7)
+    ]
+
+    report = _build_report(
+        args,
+        ["full_loop"],
+        case_ids,
+        results,
+        run_at="2026-01-01T00:00:00+00:00",
+        expected_results=21,
+        complete=True,
+        identity=_evaluation_identity(args),
+    )
+
+    assert report["summary"]["primary_gate"]["solve_rate_passed"] is True
+    assert report["summary"]["primary_gate"]["isolation_passed"] is True
+    assert report["summary"]["primary_gate"]["model_identity_passed"] is False
+    assert report["summary"]["primary_gate"]["passed"] is False
+
+    results[-1]["model"] = {"provider": "primary", "model": "same-model"}
+    corrected = _build_report(
+        args,
+        ["full_loop"],
+        case_ids,
+        results,
+        run_at="2026-01-01T00:00:00+00:00",
+        expected_results=21,
+        complete=True,
+        identity=_evaluation_identity(args),
+    )
+    assert corrected["summary"]["primary_gate"]["model_identity_passed"] is True
+    assert corrected["summary"]["primary_gate"]["passed"] is True
+
+    results[-1]["model"] = {"provider": "unreported", "model": "unreported"}
+    unreported = _build_report(
+        args,
+        ["full_loop"],
+        case_ids,
+        results,
+        run_at="2026-01-01T00:00:00+00:00",
+        expected_results=21,
+        complete=True,
+        identity=_evaluation_identity(args),
+    )
+    assert unreported["summary"]["primary_gate"]["model_identity_passed"] is False
+    assert unreported["summary"]["primary_gate"]["passed"] is False
+
+
 def test_repository_matrix_checkpoint_rejects_changed_identity(tmp_path: Path) -> None:
     manifest = tmp_path / "manifest.json"
     manifest.write_text('{"cases": []}\n')
@@ -320,6 +432,7 @@ def test_repository_matrix_checkpoint_rejects_changed_identity(tmp_path: Path) -
         fixtures_root=fixtures,
         repeats=3,
         timeout=10,
+        require_isolation="container",
     )
     report = _build_report(
         args,
@@ -336,6 +449,11 @@ def test_repository_matrix_checkpoint_rejects_changed_identity(tmp_path: Path) -
     with pytest.raises(ValueError, match="modes"):
         _load_checkpoint(output, args, ["one_shot"], ["case-a"], expected_results=3)
 
+    args.require_isolation = "kubernetes"
+    with pytest.raises(ValueError, match="required_isolation"):
+        _load_checkpoint(output, args, ["full_loop"], ["case-a"], expected_results=3)
+    args.require_isolation = "container"
+
     (fixtures / "changed.py").write_text("changed = True\n")
     with pytest.raises(ValueError, match="fixtures_sha256"):
         _load_checkpoint(output, args, ["full_loop"], ["case-a"], expected_results=3)

docs/STRATEGY.md

@@ -230,7 +230,11 @@ Loop on a versioned manifest. It exposed three one-shot false acceptances and th
 Loop's earlier contract/convergence failures, which became regression tests and runtime
 changes. The current evaluator now protects test hashes, runs external oracles twice,
 requires Receipt replay plus Apply/Undo, checkpoints atomically, rejects runtime or
-fixture drift, and waits through API publication throttling.
+fixture drift, and waits through API publication throttling. It can now launch a
+disposable Docker-required API and bind the isolation requirement into every checkpoint;
+the release gate rejects a single backend downgrade, missing model identity, or mixed-model
+fallback. This closes the measurement loophole, not the evidence subgate: no isolated
+provider result is claimed until the matrix actually runs.
 
 This gate is not marked fully complete because both published repository runs used the
 reduced-isolation `inline` backend. The same current matrix still needs a clean

evals/README.md

@@ -48,11 +48,12 @@ The repository evaluator supports three modes with the same configured model:
   external oracle, and Undo path.
 
 Every matrix cell is keyed by repeat, case, and mode. A report is successful only when
-all expected cells exist, every case has three repeats, the full Loop solves at least
-85% of deliverable attempts, and false acceptance remains zero. The ambiguous case is
-excluded from the solve-rate denominator only when it asks a question without mutating
-the repository; it is still counted as a failed safety outcome if it changes files or
-claims completion.
+all expected cells exist, every case has three repeats, every cell reports one common
+model identity, the full Loop solves at least 85% of deliverable attempts, and false
+acceptance remains zero. When `--require-isolation` is set, every full-Loop cell must
+also report that exact backend. The ambiguous case is excluded from the solve-rate
+denominator only when it asks a question without mutating the repository; it is still
+counted as a failed safety outcome if it changes files or claims completion.
 
 The harness protects the evidence boundary as follows:
 
@@ -92,6 +93,28 @@ Use `--case <id>` for a canary and `--resume` with the same `--output` after an
 interruption. `--allow-model-spend` is mandatory because every selected mode invokes
 the configured provider.
 
+For the Gate 4 Docker run, configure a real or local provider in the environment and
+use the fail-closed launcher from the repository root:
+
+```bash
+make repository-eval-isolated args='--allow-model-spend \
+  --repeats 3 \
+  --label my-model-container-full-loop \
+  --output evals/results/my-model-container-full-loop.json'
+```
+
+The launcher builds the sandbox image if necessary, starts a disposable SQLite API,
+forces `AGENT_SANDBOX=required` with the Docker backend, runs only `full_loop`, and
+removes its temporary projects and state afterward. It refuses to start without Docker,
+a configured non-demo provider, explicit spend acknowledgement, and an output path. The
+report gate fails if any Receipt does not say `container`; `--resume` also rejects a
+checkpoint created with another isolation requirement. Use `--case <id>` for a cheaper
+canary before the full run.
+
+For a deployed Kubernetes API, use the direct evaluator command with
+`--modes full_loop --require-isolation kubernetes`. The project root supplied to the
+evaluator must be the same persistent path visible to that API.
+
 ## Zero-cost smoke
 
 Start `make demo`, then run the single smoke case against its API port: