Cohort scanners: cross-transcript analysis

[trentshaines]

Cohort scanners: cross-transcript analysis

Adds cohort scanners — scanners whose input is a group of related transcripts (Sequence[Transcript]) rather than a single one — so an agent can root-cause why some attempts at the same task succeeded and others failed, and identify the chokepoints where they diverged, by comparing transcripts.

This realizes the latent Sequence[Transcript] scanner input described in design/scout.qmd that was never implemented. The change is additive — existing per-transcript scanners are unchanged.

Scope note: this is the single-process slice (Stages 0–2). Multi-process cohort execution, map-reduce for context limits, and cohort-aware UI are deferred follow-ups.

Authoring

@scanner(group_by=["task_set", "task_id"], messages="all")
def root_cause() -> Scanner[Sequence[Transcript]]:
    async def scan(cohort: Sequence[Transcript]) -> Result:
        body, refs = await transcripts_as_str(cohort)   # labels members [T1:M3], [T2:M7]…
        ...
        return Result(value=..., references=refs)        # refs carry transcript_id
    return scan

Or use the built-in comparison agent:

scan(scanners=[llm_cohort_scanner(question="Why did some attempts fail? Identify the chokepoint.")], ...)

Differing scopes (configurable)

One scanner can run at multiple scopes via distinct keys (scout.yaml):

scanners:
  rc_across_models:  { name: root_cause, cohort: { group_by: [task_set, task_id] } }
  rc_within_sample:  { name: root_cause, cohort: { group_by: [task_set, task_id, model, agent] } }

Presets: same_task_across_models, same_task_in_sample, same_task_across_agents. Grouping dimensions are existing TranscriptInfo fields (task_set, task_id, task_repeat, model, agent, source_id).

How it works

• Grouping (_cohort.py): groups the transcript index into cohorts with a stable cohort_id + members_digest; the plan is frozen into ScanSpec.cohorts so resume rebuilds the same groups. On resume, recorded cohorts skip; drifted membership re-scans with a warning.
• Execution (_scan_cohorts.py): a phase-B executor runs each cohort scanner once per cohort, reading members lazily (bounded concurrency). The per-transcript pipeline is untouched.
• Persistence: cohort results reuse the scanner=<key>/<cohort_id>.parquet buffer layout, so compaction/discovery/scan_results_df need no changes.
• Guardrails: Reference gained a nullable transcript_id (so [T1:M3] vs [T2:M3] disambiguate); Error gained cohort_id; cohort scanners are rejected from as_scorer and per-sample metrics.

Tests / verification

• New tests: cohort grouping, mixed per-transcript + cohort in one scan, per-scanner scope override, resume skip + drift re-scan, transcripts_as_str renderer, llm_cohort_scanner e2e with cross-transcript references.
• Full suite green except 4 pre-existing failures unrelated to this change (DuckDB version drift in the dev env; confirmed on a clean checkout).
• mypy and ruff clean on all new/changed files. openapi.json + generated.ts regenerated (cohort-only, additive). Frontend pnpm typecheck passes.

Notes for review

• Built/verified against this fork's base. Intended to be forward-ported to an upstream (meridianlabs-ai/inspect_scout) PR after review.
• dist/ was not rebuilt here (this PR adds no frontend runtime code; a fresh-env pnpm build would only introduce build-tool version churn). Rebuild in the canonical env before merge if desired.