Multi-agent system for integrating, analysing and representing heterogeneous dental clinical data on a patient Digital Twin, built on Gaussian Splatting with per-point/per-region clinical attributes and support for time series.
Open source · Apache 2.0 · Python ≥ 3.13
The viewer demo shows the .uos container loaded in the interactive dental twin viewer.
Dental care produces highly heterogeneous data: CBCT scans (DICOM), STL files from intraoral scanners, clinical reports as PDF and 2D photographs. That information lives fragmented in silos, one per vendor and one per clinic, which makes real longitudinal follow-up impossible and undermines the patient's sovereignty over their own health data.
Agentic Smart Health addresses this with a multi-agent architecture that autonomously organises, integrates and analyses heterogeneous dental data, projecting it onto a digital twin of the patient. The process is reversible: the system can regenerate STL files and images directly from the Digital Twin.
Several agents (workers with a single responsibility) translate heterogeneous
clinical files (DICOM, STL, PDF, photo) into a common document — the
TwinSnapshot from core-schemas — then enrich it and
materialise it so a viewer can show it; an orchestrator
(agent-orchestrator) hands out the work. The
"model" (LLM) is not a central layer: it is the brain reasoning inside one
specific agent (today only research-agent), and not every agent needs one.
📐 Full map of the 6 layers and the path the data takes (written for newcomers):
docs/architecture/multi-agent-pipeline.md§0.
The animation below is generated from a real run over the public synthetic case: raw
OBJ/DICOM/report/photo files are ingested by the agents, assembled into a
TwinSnapshot, fused and segmented, exported as STL/PLY/PNG/viewer files, and finally
packaged as a .uos container.
Regenerate the pipeline animation and its measured event log with:
uv run python scripts/demo_pipeline_readme.pyThe measured event log behind the SVG is versioned at
docs/assets/pipeline-demo-events.json. The
generated DICOM, STL, PLY and .uos files are created in /tmp and cleaned up by
default; run with --keep-workdir to inspect them.
- Python ≥ 3.13
uvinstalled on the system
Clone the repository and install every workspace dependency with a single command:
git clone https://github.com/anfaia/agentic-smart-health.git
cd agentic-smart-health
make installThat runs uv sync, which resolves and locks all dependencies (internal and external) and creates the virtual environment in .venv/.
| Command | Runs |
|---|---|
make install |
uv sync |
make hooks |
git config core.hooksPath .githooks |
make test |
uv run pytest |
make lint |
uv run ruff check |
make docs |
uv run python scripts/docs_sync.py --write |
make install also enables the repository's git hooks
(git config core.hooksPath .githooks), and the pre-commit hook does two things:
- Stops the commit if
data_guard.pyfinds third-party data staged (a PDF, a mesh, a large binary). This is the only place where that is cheap to catch: once committed, removing it means rewriting history. - Regenerates the generated blocks of the documentation — variable, script and command tables, and the agent registry — and adds them to the same commit, so the docs always travel with the change that affects them. It only touches what sits between the markers; never the prose.
If it ever gets in the way, git commit --no-verify skips it, and CI will still
flag it on the PR.
If you need to work inside the virtual environment directly:
source .venv/bin/activateOr prefix any command with uv run to execute it inside the environment without activating it:
uv run python -c "import core_schemas; print('workspace OK')"Ingestion, fusion, segmentation and the four export channels are built and
tested, and the full path input → twin → file has an integration test. The
deliverable is a .uos container: a real clinical case closes at 12 entries and
18 assets, UOS-Core + UOS-Vol conformance, 0 errors. Acquired data does not travel
inside — it is declared by its content address, with a per-slice hash for the CBCT's
397 slices — and the reference viewer opens
it in the browser without uploading anything.
What is measured, every number obtained by re-reading what was produced rather than by promising it:
| What | Measurement | On |
|---|---|---|
| Mesh reversibility | 3.8 × 10⁻⁶ mm maximum deviation, against a budget of 0.1 mm | real scan, 110,804 vertices |
| CBCT ↔ intraoral registration | 0.452 mm over the overlapping population | real patient |
| Render from the field | PSNR 102 dB · SSIM 0.99999999, byte-for-byte reproducible | twin → PLY → render cycle |
| Printable arch | 0.372 mm (p95) · bias −0.02 mm — this is not reversibility: it measures the root reconstructor against the scanned crown | the only band with two measurements of the same tissue |
Data contract — core-schemas (Pydantic v2, schema 1.6.0). The
TwinSnapshot is the common document, carrying provenance per value. Ingestion
agents are deterministic and fail-loud: they never raise, they return status and
confidence, and there is a human-in-the-loop gate on a threshold (0.7). The
orchestrator honours a budget of <60 s.
Not yet: per-pixel colour — the signal is measured, what is missing is camera
pose — and the pathology-agent. 3dgs-engine is a placeholder: reconstruction
lives in the notebooks with gsplat.
📋 The honest closing inventory — what is measured, what is unresolved, in what order to attack it, plus the milestones and the success metrics — is in
docs/cierre-mvp.md, along with what CI cannot verify because it has no GPU.
The repository is organised as a monorepo managed with uv workspaces. The root pyproject.toml declares the workspace and automatically groups every member under apps/ and packages/:
[tool.uv.workspace]
members = ["apps/*", "packages/*"]This lets each application and package keep its own pyproject.toml and independent lifecycle while sharing a single virtual environment (.venv/) at the root and a common lockfile (uv.lock). Internal dependencies resolve through workspace references (workspace = true), without going through PyPI.
agentic-smart-health/ ← workspace root
├── pyproject.toml ← uv workspace declaration
├── uv.lock ← unified lockfile
├── Makefile ← development commands
├── apps/
│ ├── agent-orchestrator/ ← orchestrator of the multi-agent system
│ ├── research-agent/ ← research agent (RAG + scientific literature)
├── packages/
│ ├── core-schemas/ ← shared Pydantic schemas (the TwinSnapshot contract)
│ ├── ingestion-agents/ ← 4 ingestion agents (mesh · cbct · report · image)
│ ├── fusion-agents/ ← geometric and semantic fusion over the twin
│ ├── analysis-agents/ ← anatomical segmentation: region_id (FDI) per Gaussian
│ ├── export-agents/ ← mesh, field and render regenerated from the twin, with the error measured
│ ├── gaussian-engine/ ← fitting anisotropic ellipsoids to the density the CBCT measured
│ ├── uos/ ← Unified Oral Scene container: the whole case with its relations declared
│ ├── tooth-aggregation/ ← aggregating per-point labels into tooth instances
│ └── 3dgs-engine/ ← placeholder (3DGS reconstruction lives today in notebooks + gsplat)
├── data/
│ └── research-agent/ ← knowledge base of the research agent
├── schemas/ ← published JSON Schema of the UOS manifest, per version (§12)
├── docs/ ← documentation (see the note below)
├── notebooks/ ← experimentation and exploration (01–09)
├── tests/ ← global test suite
├── scripts/ ← utilities: Blender render, PR auditor, dataset fetchers
└── .github/
└── workflows/ ← CI: code review agent (ai-code-reviewer)
Central orchestrator of the multi-agent system. It coordinates the agents of each pipeline phase:
- Ingestion ✅ (implemented): fires the 4
ingestion-agentsin parallel over one acquisition (STL + CBCT + report + N photos), assembles theTwinSnapshotand applies the human review gate; budget of <60 s. - Fusion ✅ (implemented):
IngestionPipeline.fuse()chains twoGeometricFusionAgentruns — scanner↔scanner registration and the IOS↔CBCT ICP, each with itsrms_error_mmand its verification status — and theSemanticFusionAgent, which hangs the report's findings off FDI codes and flags the conflict when report and geometry disagree. - Analysis 🟡 (the anatomical part, yes; the clinical part, no): the
segmentation-agentruns insidefuse(), between the two fusion stages, and fillsPipelineResult.analysis.⚠️ Its quality is measured and is the MVP's main gap: 11 of 14 teeth are discarded on anatomical grounds (docs/research/segmentacion-fdi-escaner.md). Clinical reasoning — thepathology-agent— is stillplanned, and ships with mandatory human review by design. - Export ✅ (all four channels):
export-agentsregenerates from theTwinSnapshotthe mesh as STL, the Gaussian field as PLY (in the twin's frame or in the CBCT's real millimetres) and a multi-view render as PNG by Beer-Lambert, each with its measured error — maximum and mean deviation for geometry, PSNR/SSIM for the image. The orchestrator fires them withIngestionPipeline.exportar(result, destino), and the full path input → twin → file is tested end to end intests/test_e2e.py.
It depends on core-schemas and ingestion-agents (via the workspace) so that data contracts stay shared with the rest of the system.
Interoperability with 3D Slicer and other platforms
Through open formats, not through a server. The pipeline materialises every case as
STL, PLY and PNG, plus the TwinSnapshot's own JSON, and Slicer reads all of them
natively. That is already interoperability: there is no protocol to negotiate and no
service to keep alive, and the file still opens ten years from now without us.
There used to be a slicer-mcp-server here and it has been withdrawn. It was a
directory holding a server.py of zero lines, described in this very README in the
present tense — "exposes an interface", "lets agents interact" — and unblocking it
depended on a third party confirming the call's format and direction. An empty piece we
cannot unblock ourselves is not architecture: it is an intention written where facts are
documented.
An MCP server would make sense for live, bidirectional interaction — an agent driving the Slicer session, not reading a file. Nobody has asked for that yet, and when they do it gets built. See issue #40, which is now a question for the partner rather than a component of this repository.
An autonomous research agent that searches, ingests and summarises scientific literature on 3D Gaussian Splatting, the DICOM standard and clinical regulation. Built with Python, Anthropic Claude / Ollama, Qdrant and local embeddings.
Main capabilities:
- Semantic paper search on Semantic Scholar and arXiv
- Document ingestion and indexing through RAG (Qdrant + fastembed)
- Structured report generation in Markdown
- Local execution with Ollama (free, no API key)
Run modes:
uv run python -m src.main— Claude with native tool calling (requires an API key)uv run python -m src.main_local— local Ollama (free, 100% private)
Starting corpus. The reference PDFs are not in the repository: they are
third-party binaries and the licence of many of them does not allow
redistribution. What is versioned is the inventory
(manifest.yaml: title, DOI or
arXiv ID, URL and the licence verified at the source for each document). To
materialise them:
uv run python scripts/fetch_knowledge_base.py # download what is missing
uv run python scripts/fetch_knowledge_base.py --check # check onlyA couple of publishers (Wiley, AAAI) will not serve the PDF to a script: those are
left as manual downloads and the command prints the link. The agent works without a
corpus — search_references discovers new literature — but read_directory and
index will find nothing until it has been run.
Layout:
src/main.py— CLI orchestrator with Claudesrc/main_local.py— local variant with Ollamasrc/tools.py— system tools (disk sandbox)src/rag.py— RAG engine (Qdrant + fastembed)src/references.py— paper discovery
It does not depend on core-schemas; it keeps its own internal models for RAG.
Note: this agent is a port of jeicob, adapted to fit the monorepo.
One sentence per package; the full card for each agent is in AGENTS.md.
Single source of truth for the types: Pydantic v2 schemas shared across the whole
workspace — TwinSnapshot, Provenance, per-tooth FDI observations — versioned, today
1.6.0.
The 4 ingestion agents (mesh · cbct · report · image), one per modality.
Deterministic and fail-loud, with Provenance per value, a content-addressed
ArtifactStore and EXIF discarded by construction. To add one: the add-ingestion-agent
skill.
The only family that writes output files: STL, PLY, multi-view PNG and a printable
arch. All of them measure what they produce by re-reading it. Two things that surprise
people: the field's PLY is not a 3DGS .ply, and the render does not rasterise splats —
density is radiological attenuation, not opacity, so it composites by Beer-Lambert,
which is also order-independent and therefore byte-for-byte reproducible.
The container and its manifest, the project's deliverable: an uncompressed ZIP holding
the whole case with the relations between its parts declared. The rule that holds it
up is that the measured and the inferred do not mix: inference lives only under
derived/, and a .uos with no derived/ is still valid and complete. Schema in
schemas/, format in
docs/spec/uos-format-spec-v0.2.tex.
Fusion (ADR 004): geometric registration by ICP, always declaring its rms_error_mm
and whether anyone has verified it, plus anchoring the report's findings to FDI codes,
with the conflict flagged when report and geometry disagree.
Anatomical analysis: region_id per Gaussian and the FDI → confidence map. docs/research/segmentacion-fdi-escaner.md).
Field fitting: from isotropic voxel-sized seeds to measured ellipsoids. The only
package that touches torch, and it imports it inside the function so it installs without
CUDA.
Point → tooth aggregation (instances + FDI). Deliberately free of torch: the
forward pass is the caller's business.
Placeholder. 3DGS reconstruction lives today in the notebooks with
gsplat and Blender. It gets promoted to a package once the recipe stops being
experimental.
Nine technical validation spikes (not the final system, and not clinical results) that
de-risk the architectural decisions before each link becomes an agent. They run on real,
gitignored datasets: Teeth3DS+ (01–06) and Bite2Text (07). Notebook 07 is the one
that wires the ingestion agents into the reconstruction flow, with colour taken from the
photos and a 31.5 dB holdout.
What each one validates, its scope and how to run them:
notebooks/README.md.
Every Pull Request goes through a static review guardian agent
(ai-code-review.yml). It uses no LLM: it combines
Ruff and MyPy with a bespoke architecture auditor, and reviews only the Python files the
PR touches. It publishes inline annotations and a summary comment. Architecture
violations and coverage below 80% block the merge.
On top of that, docs_sync.py checks that this documentation does
not drift from the code — cited paths, agent registry, constants, the tree above — and a
pre-commit hook aborts the commit if it tries to version clinical data.
Literature watch — the repository's only scheduled job
(literature-watch.yml): every Monday it
searches arXiv for what was published that week, reads the licence from arXiv's OAI-PMH
(it does not assume it) and opens a PR proposing new manifest entries. It does not
merge. No PDF is ever written to the runner: they are downloaded into memory to compute
sha256 and released right there.
Repository utilities (this table is generated by docs_sync.py):
| Script | What it does |
|---|---|
scripts/ablacion_recetas.py |
Training recipe ablation: what each component contributes. |
scripts/altura_corona.py |
Measures clinical crown height on the intraoral scanner mesh. |
scripts/audit_pr.py |
Architecture guardian for pull requests in the monorepo. |
scripts/blender_render_views.py |
Headless Blender multi-view renderer for intraoral meshes. |
scripts/caso_completo.py |
Runs the full pipeline over one real clinical case, stage by stage. |
scripts/composicion_cbct_ios.py |
Composes CBCT teeth and IOS gum geometry as Gaussians. |
scripts/data_guard.py |
Blocks third-party, oversized or clinical data from entering the repository. |
scripts/demo_pipeline_readme.py |
Regenerates the README pipeline animation from the synthetic case. |
scripts/desplazamiento_relativo.py |
Measures whether tooth displacement can be reported robustly. |
scripts/docs_sync.py |
Checks that documentation stays aligned with code and repository inventory. |
scripts/entrena_diente_cbct.py |
Trains the CBCT tooth segmenter against the threshold baseline. |
scripts/entrena_gs_escaner.py |
Trains real 3DGS over the scanner surface. |
scripts/entrenar_3dgs.py |
Negative experiment: trains 3DGS from a dental arch. |
scripts/eval_informes.py |
Measures how much report content reaches the shared data contract. |
scripts/fetch_knowledge_base.py |
Materialises the research agent knowledge base. |
scripts/fetch_teeth3ds.sh |
Downloads Teeth3DS+ reproducibly from the official Google Drive source. |
scripts/malla_mejorada.py |
Builds the improved STL from the UOS container and nothing else. |
scripts/metricas.py |
Computes the measured metrics promised in the project brief. |
scripts/mide_segmentacion.py |
Measures how much FDI segmentation can be discarded from a .uos file. |
scripts/promedio_y_escala.py |
Measures design questions around per-tooth registration and scaling. |
scripts/refina_3dgs.py |
Optimises the seeded Gaussian field as a 3DGS representation. |
scripts/registro_ios_cbct.py |
Measures whether the intraoral scanner and CBCT can be aligned. |
scripts/resolucion_modalidades.py |
Simulates the effective resolution of each dental modality. |
scripts/segmentar_fdi.py |
Labels each tooth in a dental arch with its FDI code. |
scripts/seguimiento_histora.py |
Measures gingival margin movement between two scans. |
scripts/verifica_contenedor.py |
Verifies that a .uos container tells the truth about itself. |
scripts/watch_literature.py |
Watches scientific literature and proposes manifest entries. |
Development tools install with uv sync --group dev (group dev: ruff, mypy). Full agent card in AGENTS.md.
Copy the example file and set the variables you need:
cp .env.example .env.env.example documents only the variables the code actually reads, together
with who uses them and what happens if they are unset. This table is generated by
scripts/docs_sync.py from the code, and CI fails if it
drifts — which is why it is not edited by hand. A — in the last column means the
call carries no default (the module may have its own fallback):
| Variable | Read in | Default |
|---|---|---|
ANTHROPIC_API_KEY |
apps/research-agent/src/main.py |
— |
ASH_PSEUDONYM_SALT |
packages/ingestion-agents/src/ingestion_agents/cbct_agent.py |
dev-salt-no-usar-en-produccion |
OLLAMA_HOST |
apps/research-agent/src/main_local.py |
http://localhost:11434 |
QDRANT_PATH |
apps/research-agent/src/rag.py |
— |
RESEARCH_AGENT_LOCAL_MODEL |
apps/research-agent/src/main_local.py |
qwen2.5:7b |
RESEARCH_AGENT_MODEL |
apps/research-agent/src/main.py |
claude-opus-4-8 |
None of them is needed to run make test. The pseudonym salt is the only one that
is a secret: without it the pipeline still runs, but the pseudonyms it emits are
not fit for patient data — and if it changes later, they stop matching the ones
already emitted.
Note: the
docs/directory is reserved exclusively for research and architecture documentation. It holds no user documentation and no usage tutorials.
docs/architecture/— design decisions, architecture diagrams and ADRs (Architecture Decision Records).docs/research/— bibliographic references and research notes on Gaussian Splatting, DICOM/STL standards, clinical interoperability and applicable regulation (GDPR, HIPAA).docs/spec/— the normative specification of the.uosformat and the project white paper, in LaTeX.
Technical documentation aimed at developers and contributors stays in this README and in each component's pyproject.toml.
| Document | Answers |
|---|---|
docs/cierre-mvp.md |
what is measured, what is unresolved and what is left for later |
docs/spec/uos-format-spec-v0.2.tex |
the format specification: what a .uos carries, how it is read, how it is extended |
docs/spec/uos-white-paper.tex |
why a new format is needed, which hypotheses were tested and with what results |
docs/research/segmentacion-fdi-escaner.md |
why FDI segmentation is not solved, with the measurement |
docs/architecture/branching-and-release-workflow.md |
how develop, main, pull requests, tags and releases are used |
docs/research/frontera-encia-desde-foto.md |
where the tooth-gum boundary actually is, and what is missing to use it |
docs/research/color-por-pieza-desde-foto.md |
the shade of each crown, and how the flash falloff is discounted without inverting it |
docs/research/segmentacion-diente-cbct.md |
how far a classifier gets on the CBCT, and where it stops getting there |
-
ANFAIA — Artificial Intelligence Non-Profit Research Association driving open-source AI solutions for global health.
-
HISTORA — the dental startup this work was carried out with.
ANFAIA Summer Grants 2026 · July – August 2026