EHR Event Extraction

This repository contains the codebase accompanying the paper LLM-Based Event Extraction from Clinical Notes: A Privacy- and Resource-Constrained Case Study on Diagnoses.

The project investigates the use of large language models to extract structured clinical events from free-text electronic health record notes and to assign temporal information to the extracted events. The focus is on diagnosis events, with evaluation conducted on both public and private clinical datasets.

The repository includes:

• preprocessing and parsing utilities for clinical EHR data
• a generic, LLM-driven clinical event extraction pipeline
• supporting resources such as medical code systems and an ontology
• experiment code and evaluation scripts used in the paper

The code is primarily intended to support the experiments and analyses reported in the paper, with an emphasis on transparency and reproducibility rather than production deployment.

Repository overview

The repository is organized around a small set of core components: data resources, reusable libraries, experiments, and auxiliary scripts.

At a high level:

• src/ contains reusable components for parsing EHR data, interacting with LLMs, and extracting structured clinical events from text.
• resources/ provides supporting material such as diagnosis code systems and an OWL ontology.
• datasets/ defines the expected directory structure for input data and generated artifacts. No clinical data is distributed.
• experiments/ contains the experiment code and evaluation logic used to produce the results reported in the paper.
• scripts/ includes helper scripts to prepare datasets and generate resource files.

The following sections describe each component in more detail, moving from shared infrastructure to experiment-specific code.

Execution environment

All experiments were run using Python 3.12.

PyTorch must be installed separately, following the official instructions at https://pytorch.org/, and configured according to the target hardware (CPU or GPU).

All remaining dependencies are listed in the requirements file.

At the time of execution, the main packages and versions were:

ollama        0.6.1
openpyxl      3.1.5
pandas        2.3.3
torch         2.9.1
torchaudio    2.9.1
torchvision   0.24.1
tqdm          4.67.1
transformers  4.57.3
tzdata        2025.3
Unidecode     1.4.0
vllm          0.13.0

Data and resources

This repository does not distribute any clinical data. It provides the code and supporting resources used to preprocess datasets and run the experiments described in the paper.

The following subsections describe each resource in more detail.

Datasets

This repository expects datasets to be placed under the datasets/ directory following a fixed structure. No clinical data is distributed.

Two datasets are supported:

• MDACE A publicly usable dataset with controlled access, providing manual diagnosis annotations linked to clinical notes derived from MIMIC-III.

  To make MDACE operational, the following files are required:

  • MDACE ICD-10 annotation JSON files from MDACE ICD-10 annotations placed under datasets/mdace/raw/
  • the NOTEEVENTS.csv file from MIMIC-III placed under datasets/mimic/raw/

• FNUX A private dataset obtained through collaboration with Aalborg University Hospital. While all parsing and preprocessing code is provided, the data itself is not publicly available.

Prepared outputs are written to datasets/<dataset>/diagnoses.json.

Dataset preparation can be run with:

python -m scripts.prepare_diagnosis_datasets

Datasets for which input data is not present are skipped without error.

Ontology

The file resources/ontologies/fnux.owl contains an OWL ontology developed to model the clinical information represented in the FNUX-based EHR data used in this project.

The ontology defines the core clinical entities, events, and relationships required to support extraction, normalization, and analysis.

At its current stage, the ontology focuses on class structure and basic properties. Detailed axioms and formal constraints are limited.

Code systems

Some datasets used in this project provide diagnosis codes without natural language descriptions or include non-diagnostic concepts within the same coding system.

To support consistent diagnosis extraction and filtering, the repository includes local representations of several diagnosis code systems with their official textual definitions.

Supported systems include:

• ICD-10
• ICD-10-CM
• ICPC-2

Code system data is generated automatically from public sources using:

python -m scripts.build_code_systems

The resulting mappings are stored under resources/code_systems/.

Diagnosis filtering logic is implemented in src/code_system.py and operates on both code structure and textual descriptions. The filtering procedure follows the methodology described in the paper and is intended as a heuristic for dataset consistency rather than a clinical gold standard.

Experiments

The experiments/ directory contains the code used to produce the experimental results reported in the paper.

Experiments are implemented as self-contained Python modules and rely on the core libraries under src/. They are designed to be reproducible given fixed model configurations and available datasets.

Diagnosis extraction

The diagnosis extraction experiment evaluates the ability of the event extraction pipeline to identify and structure diagnosis events from free-text clinical notes.

The workflow consists of:

• running the diagnosis extractor on clinical documents
• matching extracted diagnoses against ground-truth annotations
• computing name-level and temporal evaluation metrics

Datasets

The experiment is run on two datasets:

• FNUX Danish EHR documents with diagnosis codes and associated dates. This dataset supports both name-based and temporal evaluation.

• MDACE Clinical notes with manual diagnosis annotations derived from MIMIC-III. Due to limited temporal information, only name-based evaluation is performed.

Evaluation

Evaluation is performed at the event level.

Extracted diagnosis names are matched to ground-truth names using:

• lightweight string-based heuristics
• an optional LLM-based judge for ambiguous cases

Name extraction quality is reported using precision, recall, and F1 score, computed on:

• all extracted diagnoses
• a filtered subset retaining only stricter diagnosis events

For FNUX, temporal accuracy is also evaluated using:

• exact date match
• match within one week
• match within one month

LLM

Experiments are designed to run with large language models served via the Ollama framework, as implemented in src/llm.

In the experiments reported in the paper, we used MedGemma 27B, running locally via Ollama. The model is available at symptoma/medgemma.

The model can be downloaded with:

ollama pull symptoma/medgemma:27b

Running the experiment

The diagnosis extraction experiment can be executed as a Python module:

python -m experiments.diagnosis_extraction <ollama_code>

To run the experiment using the same LLM configuration as in the paper:

python -m experiments.diagnosis_extraction symptoma/medgemma:27b

Judge validation

The file resources/judge_validation.csv contains manual annotations used to assess the reliability of the LLM-based diagnosis equivalence judge.

Each row corresponds to a pair of diagnosis strings and includes binary equivalence judgments produced by the LLM and by three human annotators. These annotations are used exclusively to evaluate inter-annotator agreement, as described in the paper.

The file does not contain patient-level data, clinical notes, or identifiers.

Core libraries

The src/ directory contains the core, reusable libraries developed for this project. These components implement EHR parsing, LLM interaction, and generic clinical event extraction, and are used across experiments.

Each library is described in more detail in the following subsections.

FNUX XML parser

The src/fnux package provides an XML parser that converts EHR documents in the MedCom FNUX format into typed Python objects using a structured, mapping-driven approach.

The design emphasizes determinism and transparency:

• explicit mappings from XML elements to object fields
• no implicit inference or hidden defaults
• strict validation of structural and identity constraints

The parser assumes that each FNUX file belongs to at most one patient. This constraint was observed in the real data used in this project but is not enforced by the original FNUX specification. As a result, parsing arbitrary FNUX files that violate this assumption may raise errors.

LLM support

The src/llm module defines a minimal abstraction layer for interacting with large language models.

It provides:

• a backend-agnostic interface for text generation
• concrete backends for local inference engines
• explicit, model-specific instruction templates for instruction-tuned models

The implementation operates directly on prompt strings rather than relying on higher-level chat abstractions, allowing precise control over formatting and special tokens.

This module is intended for experimental use and evaluation, not as a general-purpose LLM framework.

Clinical event extractor

The src/extractor package implements a generic, LLM-driven pipeline for extracting structured clinical events from free-text clinical notes.

The extractor is event-agnostic. Event-specific behavior is defined through declarative event specifications that describe:

• how events are detected
• how many instances are expected
• how extracted information is structured

Extraction follows a fixed three-step process: classification, event identification, and structuring. This design allows new event types to be added without modifying the core extraction logic.

Code systems (diagnoses)

The src/code_system utilities provide:

• loading of code → description mappings generated under resources/code_systems/
• conservative filtering rules to exclude non-diagnostic concepts (e.g. symptoms, encounters, process codes)

Filtering operates on both code structure and description text and is used to enforce a stricter, cross-system notion of diagnosis events in the experiments.