agentic-shared

Shared infrastructure and evaluation harness for agentic AI projects: docker-compose stack, vendored eval package with G-Eval / retrieval@k / acceptance-rate / latency / CI regression gate, frozen dependency pins, CI workflow template, and README template.

Architecture

graph TB
    subgraph "agentic-shared (template repo)"
        direction TB
        DC["docker-compose.yml<br/>Postgres · Qdrant · Phoenix"]
        PP["pyproject.toml<br/>Frozen pin set (Python 3.11)"]
        CI[".github/workflows/ci.yml<br/>uv + ruff + pytest"]
        RT["README.template.md<br/>7-section structure"]
        MK["Makefile<br/>up · down · test · lint · smoke"]
    end

    subgraph "shared_eval/ (vendored into each project)"
        direction TB
        BASE["base.py<br/>Evaluator protocol"]
        GE["g_eval.py<br/>LLM judge → [0,1]"]
        RET["retrieval.py<br/>precision@k · recall@k"]
        ACC["acceptance.py<br/>Binary quality gate"]
        LAT["latency.py<br/>p50/p95 timer"]
        RUN["runner.py<br/>Golden JSONL × evaluators → artifact"]
        GATE["ci_gate.py<br/>Baseline vs current regression check"]
    end

    subgraph "Consumer projects"
        P1["eval-driven-model-router"]
        P2["multimodal-doc-intel-agent"]
        P3["mcp-dev-assistant"]
        P4["multi-agent-research-workbench"]
        P5["agentic-rag-platform"]
    end

    DC --> P1 & P2 & P3 & P4 & P5
    PP --> P1 & P2 & P3 & P4 & P5
    CI --> P1 & P2 & P3 & P4 & P5
    BASE --> GE & RET & ACC & LAT
    GE & RET & ACC & LAT --> RUN --> GATE

Loading

How projects consume it

Scaffolding — agentic-shared is a GitHub template repository. Each of the 5 sprint projects was created from this template, inheriting docker-compose, CI workflow, README template, .gitignore, LICENSE, and the base pyproject.toml.

Eval harness — The shared_eval/ package is vendored (copied) into each project at scaffold time. It is not a published PyPI package (deferred to a future polish window).

Pin discipline — The pyproject.toml defines the single frozen dependency set. All 5 projects inherit this base and add per-project extras on top.

Quick start

# Clone
git clone git@github.com:ghoshp83/agentic-shared.git
cd agentic-shared
cp .env.example .env   # fill in API keys for live eval

# Install
make install            # pip install -e ".[dev]"

# Start infrastructure
make up                 # postgres + qdrant + phoenix (waits for healthy)

# Run tests (mocked judge — no LLM key needed)
make test               # 32 tests, all pass

# Run non-LLM smoke
make smoke              # eval harness over 5-item golden set with mocked judge

# Lint
make lint               # ruff check

The eval harness

Evaluator protocol

Every metric implements the same contract:

from shared_eval.base import Evaluator

class MyEvaluator:
    name = "my_metric"

    def score(
        self,
        prediction: str,
        reference: str | None = None,
        context: list[str] | None = None,
    ) -> dict[str, float]:
        ...

Built-in evaluators

Evaluator	Module	What it measures
GEval	g_eval.py	LLM judge (gpt-4o-mini) scores answer quality 1-5, normalised to [0, 1]. Criterion is configurable. Judge callable is injectable for testing.
RetrievalAtK	retrieval.py	Precision and recall of retrieved context IDs vs gold IDs. Pure Python, no dependencies.
AcceptanceRate	acceptance.py	Binary quality gate — configurable predicate (default: non-empty + has citation). Optional Postgres persistence.
LatencyTimer	latency.py	Context-manager that accumulates call-level samples, reports p50/p95 quantiles.

Runner

from shared_eval import GEval, RetrievalAtK, run

result = run(
    golden_path="tests/golden/smoke_golden.jsonl",
    evaluators=[GEval(judge=my_mock), RetrievalAtK(k=5)],
    run_id="v1",
)
# result.metrics → {"g_eval": 0.75, "retrieval@5": 0.60, "recall@5": 0.85}
# Writes eval_artifact_v1.json + optional Postgres row

CI gate

python -m shared_eval.ci_gate \
    --baseline eval_artifact_baseline.json \
    --current  eval_artifact_latest.json \
    --tol 0.05
# Exits 0 on pass, 1 on regression beyond tolerance
# Metrics with "_ms", "cost_usd", or "latency" → lower is better
# All others → higher is better

Why this stack

Choice	Rationale
Gemini 2.5 Pro	Reasoning LLM — strong function-calling, competitive cost, consistent across the sprint
gpt-4o-mini	Eval judge — fast, cheap, good calibration for Likert scoring
Qdrant	Vector DB — binary quantization, hybrid search, mature Python client
LangGraph 0.3	Agent framework — explicit state graph, checkpointing, human-in-the-loop
Arize Phoenix	Observability — trace every LLM call, latency histograms, zero config
Postgres 16	Eval scoreboard persistence, state logging, FTS where needed
Python 3.11	Pinned for reproducibility across all 5 projects
uv	Fast dependency resolver — handles the full graph that pip's backtracker cannot

Infrastructure services

Service	Image	Port	Healthcheck
Postgres	postgres:16	5432	pg_isready
Qdrant	qdrant/qdrant:v1.12.4	6333	TCP probe
Phoenix	arizephoenix/phoenix:version-4.21.0	6006	HTTP GET /

Honest disclaimer

Component	Status	Detail
Eval harness (shared_eval)	Real	Protocol, runner, CI gate, all built-in evaluators are functional. 32 tests pass with mocked judge.
Docker-compose stack	Real	Postgres + Qdrant + Phoenix start and pass healthchecks. Verified locally.
G-Eval live scoring	Requires API key	The _default_judge calls gpt-4o-mini via the OpenAI SDK. Without OPENAI_API_KEY, tests use an injected mock. Live scoring verified in consumer projects.
Postgres persistence	Requires running service	runner._persist and acceptance._persist write to eval_scoreboard when POSTGRES_DSN is set. Falls back silently when unavailable.
PyPI package	Deferred	shared_eval is vendored (copied), not pip-installable. PyPI publication is planned for the post-sprint polish window.
Template repo scaffolding	Illustrative	GitHub's "Use this template" feature works, but the consumer projects were scaffolded manually during the sprint. Cookiecutter parameterisation is deferred.

This repository is the shared infrastructure backbone for a portfolio of 5 agentic AI projects. Components marked above require API keys or running services for full functionality. All core logic is tested with mocked dependencies.

Project structure

agentic-shared/
├── shared_eval/           # Vendored eval harness
│   ├── __init__.py
│   ├── base.py            # Evaluator protocol + EvalResult
│   ├── g_eval.py          # LLM judge (gpt-4o-mini)
│   ├── retrieval.py       # retrieval@k / recall@k
│   ├── acceptance.py      # Binary quality gate
│   ├── latency.py         # p50/p95 timer
│   ├── runner.py          # Golden JSONL × evaluators → artifact
│   └── ci_gate.py         # Baseline vs current regression gate
├── tests/
│   ├── golden/            # Golden sets for testing
│   │   └── smoke_golden.jsonl
│   ├── conftest.py        # Shared fixtures (mock judge)
│   └── test_shared_eval.py
├── scripts/
│   ├── initdb.sql         # Postgres schema (eval_scoreboard)
│   └── smoke.py           # Non-LLM smoke test
├── docker-compose.yml     # Postgres + Qdrant + Phoenix
├── Makefile               # up / down / test / lint / smoke
├── pyproject.toml         # Frozen pin set (source of truth)
├── README.template.md     # 7-section README structure for consumer projects
├── .github/workflows/ci.yml
├── .gitignore
├── .env.example
└── LICENSE (MIT)

License

MIT