RxBench-SOL

A medication-safety action benchmark for language models

RxBench-SOL evaluates whether a language model chooses the right response action for medication-safety questions. Instead of asking only whether a model knows a medical fact, it tests whether the model should:

Action	Meaning
ANSWER	The question has enough information for a normal low-risk response.
CLARIFY	Safety-critical context is missing and should be obtained first.
CAUTION	A response is possible, but it needs warning, monitoring, avoidance guidance, or clinician/pharmacist advice.
URGENT	Immediate escalation is indicated, such as poison control, emergency symptoms, severe overdose risk, self-harm, or intentional misuse.

RxBench-SOL evaluates benchmark-level medication-safety response behavior. It does not establish that a model is safe for clinical deployment or patient use.

Why This Exists

Medication questions are often underspecified. A user may omit dose, timing, age, weight, pregnancy status, kidney function, interacting medicines, or symptoms. A model can know the relevant medical fact and still respond unsafely by:

• answering before necessary context is known
• missing an interaction, contraindication, duplicate ingredient, or threshold risk
• underreacting to overdose or emergency symptoms
• escalating harmless questions unnecessarily
• producing malformed outputs that break downstream safety systems

RxBench-SOL makes those action-policy failures measurable.

flowchart LR
  A["Medication question"] --> B["Model prediction"]
  B --> C["ANSWER / CLARIFY / CAUTION / URGENT"]
  C --> D["Action-level safety scoring"]
  D --> E["Accuracy, critical unsafe, under-caution, over-caution, parse failure"]

Loading

Public Release

Primary benchmark version: RxBench-SOL v0.8.1 Audited

Property	Value
Cases	1,100
Split	pilot/evaluation
Label status	source-backed, builder-audited
Clinical validation	not clinician-validated
Validation errors	0
Patient records	none; cases are constructed examples
Recommended use	benchmark-level model evaluation

v0.8 was the initial hard-clean evaluation. A benchmark-wide audit identified objective prompt/metadata contradictions, producing v0.8.1. All listed reference systems were rerun from scratch on v0.8.1. The historical v0.8 set is not silently replaced; v0.8.1 is the recommended public evaluation version.

Dataset Composition

Action Distribution

Action	Cases
ANSWER	196
CLARIFY	276
CAUTION	386
URGENT	242

Medication-Safety Domains

Domain	Cases
dosing error	201
drug-drug interaction	157
renal/hepatic	133
pediatric	132
pregnancy/lactation	128
overdose/poison control	96
over-referral trap	80
duplicate ingredient	51
geriatric	53
safe routine question	24
OTC stacking	15
contraindication	13
allergy conflict	11
brand/generic confusion	3
missed-dose/double-dose	3

Source Backing

Cases are constructed benchmark examples, not real patient records. Labels are backed by public medication-safety sources such as DailyMed, MedlinePlus, FDA safety communications, LactMed, Poison Control, and AGS Beers Criteria.

“Source-backed” means the benchmark label points to documented public sources. It does not mean clinician validation, clinical trial validation, or deployment safety.

Metrics

The reference scorer reports:

• native action accuracy
• safety-weighted score
• critical unsafe rate
• under-caution rate
• over-caution rate
• parse-failure rate
• confusion matrix
• per-action recall
• per-bucket accuracy
• Wilson confidence intervals where appropriate

Invalid, duplicate, missing, or extra predictions are handled explicitly. Parse failures are never silently dropped.

Quick Start

python -m pip install -e .

python -m rxbench_sol.validate \
  data/rxbench_sol_v0_8_1_audited/cases.jsonl \
  --source-manifest-csv data/rxbench_sol_v0_8_1_audited/source_manifest.csv

python -m rxbench_sol.evaluate \
  --cases data/rxbench_sol_v0_8_1_audited/cases.jsonl \
  --predictions data/examples/sample_predictions.jsonl \
  --output-dir results/example

Prediction Format

Prediction JSONL rows must include case_id and a predicted action.

{"case_id": "rxbench-sol-v08-0001", "predicted_action": "caution"}

The evaluator also accepts A/B/C/D aliases and parseable raw_output, but explicit predicted_action is recommended.

Reporting Checklist

When reporting results, include:

• exact model identifier and snapshot/version
• prompt and chat template
• reasoning or thinking configuration
• tools enabled or disabled
• temperature, seed, and maximum output tokens
• parsing method
• benchmark version
• all safety metrics and parse failures
• whether the model was trained on benchmark data

Public-Health Context

Medication-safety response behavior is a public-health concern because medication questions occur at scale and often omit key context. Evaluation should measure both underreaction and overreaction. See PUBLIC_HEALTH_CONTEXT.md.

Comparison With Existing Benchmarks

RxBench-SOL is complementary to medical knowledge and broad health-conversation benchmarks. Its distinctive focus is medication-safety action selection under missing context, threshold uncertainty, interaction risk, and escalation pressure. See docs/benchmark_comparison.md.

Intended Use

Use RxBench-SOL for benchmark-level medication-safety action evaluation. Do not use it to certify clinical deployment, diagnose patients, replace professional triage, or claim that a model is safe for patient use.

Limitations

RxBench-SOL is source-backed and builder-audited, not clinician-validated. It is a constructed benchmark, not a clinical trial, medical-device validation, or patient-safety certification. Public release also creates contamination risk for future model training.

License

• Code: Apache-2.0
• Dataset and documentation: CC BY 4.0, subject to NOTICE

Citation

See CITATION.cff.