Project Memory

Verified project handoff, not repository chat.

When the developer leaves, the project keeps its knowledge.

Project Memory is a private AI project-handoff workflow for engineers inheriting an existing software service. It turns scattered repository evidence into a persistent Verified Handoff Ledger, independently checks important claims, surfaces contradictions and unknowns, and calculates a deterministic Handoff Readiness Score before ownership transfer.

Why this exists

A repository rarely contains the full mental model of the engineer handing it over. Critical context is fragmented across code, README files, configuration, ADRs, test/build output, handoff notes, known issues, and unfinished work. A new owner can easily follow stale documentation or infer an undocumented reason.

Project Memory is designed around one rule: unknown is better than invented.

What is different from "chat with your repo"

Project Memory creates reusable, typed project state rather than answering directly from arbitrary chunks:

Repository + Docs + Notes + Logs
             │
             ▼
    Deterministic Ingestion
   hashes · source types · commit
             │
             ▼
       Handoff Analyst
        candidate claims
             │
             ▼
    Independent Verifier
             │
             ▼
     VERIFIED HANDOFF LEDGER
 verified · conflicted · unknown
 unsupported · stale · failed
          ┌──┴───────────┐
          ▼              ▼
 Deterministic       Evidence-backed
 Readiness Score          Q&A

Machine-consumed agent state uses structured output. Exact evidence paths are grounded by deterministic application code rather than depending on model prose formatting. Persistent SQLite ledgers invalidate claims as STALE when cited source hashes change.

Final measured result

The final comparison uses the same 10 synthetic projects / 20 frozen handoff tasks, the same project evidence, the same Groq openai/gpt-oss-120b model, and the same deterministic-v3 scorer for both systems. Gold rubrics are evaluator-only.

Metric	Generic baseline	Project Memory	Change
Critical Handoff Task Success (CHTS)	0.700 (14/20)	1.000 (20/20)	+0.300
Required fact recall	0.968	1.000	+0.032
Evidence support	0.941	1.000	+0.059
Unknown recall	1.000	1.000	0.000
Conflict recall	0.750	1.000	+0.250
Forbidden-claim task rate	0.050	0.000	-0.050
Average answer latency	1.175 s	6.170 s	+4.995 s

Primary outcome: +42.9% relative CHTS improvement (14/20 → 20/20).

The latency increase is intentional and reported: Project Memory performs reusable project-level analysis and independent verification before answering. See docs/evaluation.md.

Quick start

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -U pip
pip install -e '.[dev]'
pytest -q

Verify the frozen benchmark:

python -m evaluation.benchmark_lock
python -m evaluation.validate_benchmark

Run the local deterministic demo:

project-memory demo

Demo dashboard

The submitted recorded evaluation artifacts drive a judge-friendly dashboard:

uvicorn project_memory.api:app --host 127.0.0.1 --port 8000

Open http://127.0.0.1:8000/.

The dashboard shows the baseline/final comparison and lets a judge inspect per-project readiness, verified claims, evidence paths, stored answers, and trajectory counts. API docs remain available at /docs.

Reproduce the baseline and final workflow

The LLM adapter is OpenAI-compatible. The recorded experiment used Groq's free-tier endpoint. Never place a key in source code or commit history.

read -rsp 'Groq API key: ' GROQ_API_KEY; echo
export LLM_API_KEY="$GROQ_API_KEY"
export LLM_BASE_URL="https://api.groq.com/openai/v1"
export LLM_MODEL="openai/gpt-oss-120b"
export LLM_TIMEOUT_SECONDS="120"

Baseline:

python -m evaluation.run_baseline \
  --output evaluation/results/baseline/reproduction \
  --request-delay 10 --max-retries 8 --retry-base-seconds 12

Project Memory:

python -m evaluation.run_project_memory \
  --output evaluation/results/project-memory/reproduction \
  --request-delay 20 --max-retries 8 --retry-base-seconds 12

Score either answer file with the same evaluator:

python -m evaluation.evaluate_run <answers.jsonl> --output <metrics-v3.json>

For exact commands, expected artifacts, runtime/cost notes, and resume instructions see docs/reproduction.md.

Verify the recorded experiment

python -m evaluation.verify_final_v43
python scripts/prepare_submission.py

The final lock protects the immutable raw baseline/final answers, run manifests, and v3 metric files. Older evaluator generations are intentionally retained as audit evidence.

Agent trajectories

Every evaluated project stores a JSONL trajectory containing the instructions, inputs, output state, model usage/latency, and final answers for all three reasoning agents:

• handoff_analyst
• independent_verifier
• handoff_answer

python scripts/prepare_submission.py copies representative judge-friendly trajectories into trajectories/ while retaining the full run.

Improvement story

The most important improvements came from observed failures, not from adding more agents:

1. A fair generic baseline scored 0.700 CHTS under the final scorer.
2. Independent verification + persistent typed project state improved multi-evidence handoff reasoning.
3. Removed approach: prompt-only "return JSON" contracts; a malformed verifier response crashed a full run.
4. API-enforced structured output made agent-to-agent state reliable.
5. Removed approach: model-only citation formatting; the model retrieved the correct ADR but omitted its exact path.
6. Deterministic evidence-path grounding made provenance a product guarantee.
7. Final v3 scoring distinguishes a forbidden assertion from correctly reporting/negating a stale source claim.

The complete audit trail, including failed experiments and retained v1/v2 results, is in docs/improvement-changelog.md.

Hot take

The biggest reliability gain did not come from adding more agents. It came from moving trust boundaries out of prose: machine-consumed state is schema-constrained, citations are deterministic, and claims are independently verified.

Security and scope

• Benchmark fixtures use synthetic project data.
• Secret-looking values are masked during ingestion.
• API keys stay outside repository content and submitted artifacts.
• Project Memory does not autonomously deploy, modify production systems, or take consequential actions.
• Gold evaluation files are never exposed to agents.

Submission docs

• docs/architecture.md — design and trust boundaries
• docs/evaluation.md — benchmark and measured result
• docs/improvement-changelog.md — experiment history
• docs/reproduction.md — clean-environment reproduction
• docs/demo-script.md — ≤5-minute video plan
• trajectories/README.md — trajectory format

Coding-agent disclosure

ChatGPT (OpenAI) was used as the coding agent during development.

• Tool disclosure
• Coding-agent development trace
• Runtime-agent trajectories

The coding-agent trace is sanitized to exclude credentials and records visible instructions, commands, failures, tool outcomes, human checkpoints, and engineering decisions.