OHDSI Study Design Assistant

This repository is building an agent-style interface for common OHDSI study design tasks. The current implementation is strongest in two areas:

• phenotype recommendation for target and outcome cohort selection
• Keeper-assisted concept generation, profile extraction, and row adjudication for phenotype validation

The project separates orchestration from deterministic tooling:

• acp_agent/: ACP server that exposes the flow endpoints and handles LLM orchestration
• mcp_server/: MCP server that exposes retrieval, prompt, vocabulary, and Keeper tools
• core/: pure validation and business logic shared by ACP and MCP
• R/OHDSIAssistant/: R-side shell for the Strategus incidence workflow

What Problems This Solves

Researchers often have three immediate bottlenecks when designing an OHDSI study:

• finding a reasonable starting phenotype definition for a study intent
• refining or validating that phenotype before using it in downstream analyses
• moving from phenotype selection into a reproducible study workflow

This repo addresses those bottlenecks by combining:

• phenotype retrieval from an indexed phenotype library
• constrained LLM ranking or critique with deterministic validation
• Keeper-oriented tooling for concept generation, OMOP profile extraction, and row-level adjudication using sanitized summaries only
• an R shell that turns selected cohorts into a reproducible Strategus incidence workflow

At no point should raw row-level patient data be sent directly to an LLM.

What Is Usable Now

1. Phenotype Recommendation

Implemented flow:

1. Retrieve phenotype candidates with MCP phenotype_search
2. Build the prompt and schema with MCP phenotype_prompt_bundle
3. Rank candidates with an OpenAI-compatible LLM
4. Validate and filter results in core
5. Return diagnostics and explicit fallback metadata if the LLM output is unusable

Related implemented flows:

• phenotype_recommendation
• phenotype_recommendation_advice
• phenotype_improvements
• phenotype_intent_split
• concept_sets_review
• cohort_critique_general_design

This same recommendation path is already wired into the R Strategus incidence shell for target/outcome selection.

Primary references:

• docs/PHENOTYPE_RECOMMENDATION_DESIGN.md
• docs/STRATEGUS_SHELL.md
• docs/INCIDENCE_WORKFLOW.md

2. Keeper-Assisted Phenotype Validation

This is the other strong implemented story. It covers concept generation through case-review input preparation and row adjudication.

Implemented workflow:

1. Generate Keeper-oriented concept sets with keeper_concept_sets_generate
2. Extract OMOP-backed Keeper profiles with keeper_profiles_generate
3. Convert those profiles into review rows
4. Sanitize each row before any LLM call
5. Run phenotype_validation_review to adjudicate a single review row as yes, no, or unknown

Current characteristics:

• concept generation can use Hecate-backed, generic-search, or DB-backed vocabulary tooling
• profile extraction is deterministic only and does not call an LLM
• downstream adjudication is constrained by fail-closed sanitization and a small label set

Primary references:

• docs/KEEPER_INTERFACE_SPEC.md
• docs/PHENOTYPE_VALIDATION_REVIEW.md
• docs/TESTING.md

End-To-End Workflows

Workflow A: Go from study intent to suggested phenotypes

Use this when you need a defensible starting cohort definition for a target or outcome.

1. Start MCP and ACP
2. Call phenotype_recommendation with a study intent
3. Review returned candidates and diagnostics
4. If needed, call phenotype_recommendation_advice for next-step guidance
5. Optionally call phenotype_improvements on a selected cohort
6. If you are working in R, continue through runStrategusIncidenceShell()

Workflow B: Go from clinical event to keeper-assisted validation review

Use this when you need a practical validation loop around a phenotype.

1. Call keeper_concept_sets_generate for the phenotype of interest
2. Approve the concept sets you want to use for extraction
3. Call keeper_profiles_generate against your OMOP data
4. Take one generated rows[] entry at a time
5. Send the sanitized row to phenotype_validation_review
6. Repeat row adjudication as needed to review more sampled cases

Quickstart

Install

pip install -e .

Start MCP over HTTP

export MCP_TRANSPORT=http
export MCP_HOST=127.0.0.1
export MCP_PORT=8790
export MCP_PATH=/mcp
study-agent-mcp

Start ACP

export STUDY_AGENT_MCP_URL="http://127.0.0.1:8790/mcp"
export STUDY_AGENT_HOST=127.0.0.1
export STUDY_AGENT_PORT=8765
study-agent-acp

If you want LLM-backed phenotype flows, also set an OpenAI-compatible endpoint:

export LLM_API_KEY=<YOUR_KEY>
export LLM_API_URL="<URL_BASE>/api/chat/completions"
export LLM_MODEL=<MODEL_NAME>

This has been tested with Open webui, with locally hosted models, and LLM Shim with access to cloud services (tested with openai and bedrock models) and an embedding model serviced using the HugginFace Text Embedding Interface service.

If you want phenotype retrieval, you also need an indexed phenotype library. See docs/PHENOTYPE_INDEXING.md.

Minimal Examples

Phenotype recommendation

curl -s -X POST http://127.0.0.1:8765/flows/phenotype_recommendation \
  -H 'Content-Type: application/json' \
  -d '{"study_intent":"Identify clinical risk factors for older adult patients who experience an adverse event of acute gastrointestinal bleeding","top_k":20,"max_results":10,"candidate_limit":10}'

Keeper concept generation

curl -s -X POST http://127.0.0.1:8765/flows/keeper_concept_sets_generate \
  -H 'Content-Type: application/json' \
  -d '{"phenotype":"Gastrointestinal bleeding",
       "domain_keys":["doi","alternativeDiagnosis","symptoms"],
       "candidate_limit":5,
       "include_diagnostics":true
       }'

Keeper row adjudication

curl -s -X POST http://127.0.0.1:8765/flows/phenotype_validation_review \
  -H 'Content-Type: application/json' \
  -d '{
    "disease_name": "Gastrointestinal bleeding",
    "keeper_row": {
      "age": 44,
      "gender": "Male",
      "visitContext": "Inpatient Visit",
      "presentation": "Gastrointestinal hemorrhage",
      "priorDisease": "Peptic ulcer",
      "priorDrugs": "celecoxib",
      "afterDrugs": "naproxen"
    }
  }'

Where To Go Next

• Installation, smoke tests, and provider-specific examples: docs/TESTING.md
• Implemented service inventory: docs/SERVICE_REGISTRY.yaml
• Docker setup: see compose.yaml and .env.example
• ACP and MCP component details: acp_agent/README.md, mcp_server/README.md

Contributing

• Open an issue or discussion if a workflow is unclear or under-documented
• Submit PRs that tighten the implemented workflow docs before adding new service claims
• Join the discussion on the OHDSI Forums

Roadmap

Near-term priorities:

• strengthen phenotype recommendation and improvement workflows for study design and Strategus handoff
• expand Keeper-assisted concept generation and profile-review workflows for phenotype validation
• improve researcher-facing workflow documentation, smoke tests, and deployment guidance

Active expansion areas:

• data-quality interpretation tied to study intent
• more phenotype authoring support beyond recommendation and improvement
• broader study-design critique and cohort authoring services

For the broader future-service catalog, see docs/ROADMAP.md.

What Remains Experimental

The repository still contains broader plans that are not the main implemented story yet. Treat these as exploratory or partial unless the docs for a specific flow say otherwise:

• generalized protocol-writing and critique services
• broader data-quality interpretation services
• wider cohort authoring and design-review service families beyond the currently implemented lint/recommendation paths
• expansion toward a larger study-agent service catalog

The planned-service inventory in older docs should not be read as "fully available now".