Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
7d132e5
isolate human-reviewed conversations
jgieringer Jun 3, 2026
9ede7c5
elaborate on data used in preprint
jgieringer Jun 11, 2026
85dd6c6
adding new architecture file
luca-belli Jul 9, 2026
937e42c
adding new architecture file
luca-belli Jul 9, 2026
29e3c49
new architecture for entry-point
luca-belli Jul 10, 2026
3151059
Resolve PR #170 review feedback: config/CLI design, naming scheme, mi…
luca-belli Jul 16, 2026
112a934
Harden migration plan: fix phase-ordering bugs, add stable-interfaces…
luca-belli Jul 20, 2026
c008c42
Fix 8 critical gaps in the migration plan found under review
luca-belli Jul 20, 2026
2db14bc
Flag that generation regression testing needs a two-part approach
luca-belli Jul 20, 2026
ec4ecf6
Move the engine-testing caution from Phase 1 to Phase 5, where it bel…
luca-belli Jul 20, 2026
4293fa3
Fix 5 more gaps: dropped script, downstream naming break, manifest/co…
luca-belli Jul 20, 2026
7a68de3
Rename packages to match subcommand names; remove stale planning docs
luca-belli Jul 20, 2026
57d1659
fix: resolve architecture-doc inconsistencies and stale CODEOWNERS en…
luca-belli Jul 21, 2026
d29f6c4
fix: resolve architecture-doc inconsistencies and stale CODEOWNERS en…
luca-belli Jul 21, 2026
8d2f4e5
Add storage abstraction (Phase S, orthogonal to the rest of the migra…
luca-belli Jul 21, 2026
8954957
Propagate spine hardening: sha canonicalization, --target shorthand,
luca-belli Jul 21, 2026
0235289
Add ARCHITECTURE-SPINE.md: terse, AD-numbered invariants contract
luca-belli Jul 21, 2026
4e03d63
Mark ARCHITECTURE-SPINE.md as final
luca-belli Jul 21, 2026
80116f1
Add Phase O: firm up OpenSpec adoption from a maybe into an orthogona…
luca-belli Jul 21, 2026
7ea94c0
Resolve remaining PR #170 review threads: chatbot flag, path resoluti…
luca-belli Jul 27, 2026
096014a
Sync architecture spine with today's PR #170 review resolutions
luca-belli Jul 27, 2026
b0e91ea
Add best-guess caveat to architecture.md and the spine
luca-belli Jul 27, 2026
b0d2502
Add statistical semantic-similarity tier to Phase 5 testing
luca-belli Jul 27, 2026
6205236
Merge pull request #159 from SpringCare/mv-human-validated-assets
luca-belli Jul 28, 2026
5a26e8c
Rename generation.models to generation.user to remove field-name ambi…
luca-belli Jul 28, 2026
e40e464
Merge remote-tracking branch 'origin/main' into feat/architure.md
luca-belli Jul 28, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 17 additions & 0 deletions .github/CODEOWNERS
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# Architecture and domain packages require maintainer review.
/docs/architecture.md @SpringCare/vera-mh-maintainers
Comment thread
luca-belli marked this conversation as resolved.
/docs/vera-cli-use-cases.md @SpringCare/vera-mh-maintainers
/llm_clients/ @SpringCare/vera-mh-maintainers
/judge/ @SpringCare/vera-mh-maintainers
/generate/ @SpringCare/vera-mh-maintainers
/run_pipeline.py @SpringCare/vera-mh-maintainers
/openspec/ @SpringCare/vera-mh-maintainers
/pyproject.toml @SpringCare/vera-mh-maintainers

# Stable interfaces -- changing these requires a design doc (see docs/architecture.md#stable-interfaces-agent-coding-optimization)
/llm_clients/llm_interface.py @SpringCare/vera-mh-maintainers
/utils/role.py @SpringCare/vera-mh-maintainers
/utils/naming.py @SpringCare/vera-mh-maintainers
# /workers/queue.py, /utils/config_schema.py, and /storage/storage_backend.py will be
# added here once each file exists (Phase 5, Phase 3, and Phase S respectively --
# see docs/architecture.md#migration-from-current-layout)
54 changes: 36 additions & 18 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,19 +29,24 @@ cp .env.example .env # Add API keys (ANTHROPIC_API_KEY, OPENAI_API_KEY, et

| Area | Key paths | When to edit |
|------|-----------|--------------|
| **Generation** | `generate.py`, `generate_conversations/` | Conversation simulation, turns, personas |
| **Judging** | `judge.py`, `judge/` | Rubric scoring, TSV output, question navigation |
| **CLI** | `vera.py` | Subcommands: generate, judge, score, pool, pipeline |
| **Generation** | `generate_conversations/` | Conversation simulation, turns, personas |
| **Judging** | `judge/` | Rubric scoring, TSV output, question navigation |
| **LLM providers** | `llm_clients/`, `llm_clients/llm_factory.py` | New models, custom HTTP/API providers |
| **Pipeline** | `run_pipeline.py`, `scripts/` | End-to-end generate → judge → score workflows |
| **Pipeline helpers** | `scripts/` | Pooling and automation until absorbed into `vera pool` |
| **Data** | `data/` (personas, rubrics) | Evaluation inputs (committed) |
| **Output** | `output/` (gitignored) | Generated transcripts, evaluations, logs |
| **Config** | `utils/model_config_loader.py`, `llm_clients/config.py` | Model name resolution, API keys |
| **Shared utils** | `utils/` | Naming, logging, conversation layout |

**Entry points:** `generate.py` (simulate), `judge.py` (evaluate), `run_pipeline.py` (full workflow), `judge/score.py` (scoring/visualization).
**Entry point (target):** `vera.py` subcommands only. Legacy scripts (`generate.py`, `judge.py`, `run_pipeline.py`) are deleted entirely at the end of Phase 1 of the migration (not the full 6-phase migration) — see [docs/architecture.md](docs/architecture.md#migration-from-current-layout).

**Temporary experiments:** `tmp_tests/` (not committed). **Permanent tests:** `tests/`.

## Architecture compliance

Read [docs/architecture.md](docs/architecture.md) before structural changes. See its [ESCALATE section](docs/architecture.md#escalate-stop-and-ask) for when to stop and ask, and [Enforcement](docs/architecture.md#enforcement) for the pre-push verification commands.

## Testing

The project uses [pytest](https://docs.pytest.org/) with unit and integration tests under `tests/`. Coverage is enforced (`--cov-fail-under=30` in `pyproject.toml`).
Expand Down Expand Up @@ -76,28 +81,39 @@ uv run pytest tests/integration/

## Key Commands

Target CLI (`vera.py` — not implemented yet; use legacy commands below until Phase 1 of the migration completes):

```bash
# End-to-end pipeline (preferred for full workflows)
uv run python run_pipeline.py \
# End-to-end pipeline (target)
uv run python vera.py pipeline \
--user-agent claude-sonnet-4-5-20250929 \
--provider-agent gpt-4o \
--runs 1 \
--turns 10 \
--judge-model claude-sonnet-4-5-20250929 \
--max-personas 5

# Generate conversations only
uv run python generate.py \
-u claude-sonnet-4-5-20250929 \
-p gpt-4o \
-t 6 -r 1
# Generate / judge / score (target)
uv run python vera.py generate -u claude-sonnet-4-5-20250929 -p gpt-4o -t 6 -r 1
uv run python vera.py judge -f output/{YOUR_P_RUN}/ -j claude-sonnet-4-5-20250929
uv run python vera.py score -r output/{YOUR_P_RUN}/evaluations/{YOUR_J_RUN}/results.csv
```

Legacy (current implementation):

```bash
uv run python run_pipeline.py \
--user-agent claude-sonnet-4-5-20250929 \
--provider-agent gpt-4o \
--runs 1 \
--turns 10 \
--judge-model claude-sonnet-4-5-20250929 \
--max-personas 5

# Judge/evaluate an existing generation run
uv run python judge.py \
-f output/{YOUR_P_RUN}/ \
-j claude-sonnet-4-5-20250929
uv run python generate.py -u claude-sonnet-4-5-20250929 -p gpt-4o -t 6 -r 1
uv run python judge.py -f output/{YOUR_P_RUN}/ -j claude-sonnet-4-5-20250929

# Recommended published-score profile (scripted)
# Recommended published-score profile (scripted; legacy)
./scripts/run_recommended_vera_pipeline.sh <provider-agent-model>

# Development
Expand All @@ -119,7 +135,7 @@ Use dated model IDs (e.g. `claude-sonnet-4-5-20250929`) as in README; shorthand
- **Formatting:** `uv run ruff format .`
- **Linting:** `uv run ruff check .`
- **Type checking:** `uv run pyright` (basic mode)
- **Pre-commit:** `pre-commit install` — see `docs/pre-commit-hooks.md`
- **Pre-commit:** `pre-commit install` — see [docs/pre-commit-hooks.md](docs/pre-commit-hooks.md)
- Configuration: `pyproject.toml`

## Git Conventions
Expand Down Expand Up @@ -151,7 +167,8 @@ One canonical home per concern — cross-link, don't copy paragraphs.

| Doc | Audience | Use for |
|-----|----------|---------|
| [README.md](./README.md) | Humans | Setup, CLI usage, output layout, detailed architecture |
| [README.md](./README.md) | Humans | Setup, CLI usage, output layout |
| [docs/architecture.md](./docs/architecture.md) | Humans and agents | Target architecture, invariants, layer model |
| **AGENTS.md** (this file) | All coding agents | Style, architecture map, testing, key commands, git conventions |
| [CLAUDE.md](./CLAUDE.md) | Claude Code only | Slash commands, `.claude/` maintenance |
| [docs/](./docs/) | Humans and agents | Topic deep dives (see links below) |
Expand All @@ -162,6 +179,7 @@ One canonical home per concern — cross-link, don't copy paragraphs.

### Links

- **Architecture:** [docs/architecture.md](docs/architecture.md)
- **Setup, pipeline, output layout:** [README.md](./README.md)
- **Custom LLM providers:** [docs/evaluating.md](./docs/evaluating.md)
- **Judge behavior:** [docs/judge.md](./docs/judge.md)
Expand Down
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -429,7 +429,9 @@ VERA-MH simulates realistic conversations between Large Language Models (LLMs) f

## Architecture

### Core Components
See **[docs/architecture.md](docs/architecture.md)** for the target layer model, invariants, and single CLI orchestrator (`vera.py`). Below is a quick module reference; CLI usage and output layout are in the sections above.
Comment thread
luca-belli marked this conversation as resolved.

### Core Components (current implementation)

- **`generate.py`**: Main entry point for conversation generation with configurable parameters
- **`judge.py`**: Main entry point for evaluating conversations using LLM judges
Expand Down
4 changes: 0 additions & 4 deletions TODO

This file was deleted.

Loading