@MatiPinto03 — a short history of NLP/evaluation on regulatory drug labels.
TL;DR: From rule-based label NLP → biomedical transformers → LLMs (capable but hallucination-prone) → today's evaluation science (faithfulness, groundedness, LLM-as-judge). OpenRegEval is an evaluation tool in that last wave.
1. Origins
Drug labels were standardized as SPL (Structured Product Labeling) XML and surfaced via DailyMed/openFDA. Early mining turned labels into structured knowledge — e.g. SIDER side-effect resource (Kuhn, 2016).
2. Key milestones
- Dictionary/rule NLP → ML → biomedical transformers: BioBERT (2019), PubMedBERT.
- LLMs (GPT-3/4, 2020–2023) raised capability and hallucination risk on high-stakes text.
- Evaluation science matured: groundedness/faithfulness metrics, RAG (Lewis 2020), FActScore (2023), and LLM-as-judge patterns.
- Regulators engaged: FDA's AI/ML SaMD action plan (2021) and subsequent draft guidances on AI in drug development (2023–2025).
3. Current state
The active problem is benchmarking LLM outputs against authoritative label text — faithfulness scoring with the supporting snippet shown, human-in-the-loop adjudication. That's exactly OpenRegEval's scorecard model.
4. Open challenges & where it's heading
Reliable ground truth, automated hallucination detection, structural variability across labels/agencies, and auditability acceptable to regulators. Frontier: standardized eval frameworks and regulator-grade traceability.
This is a short orientation review, not exhaustive — years/citations are approximate, so verify primary sources before quoting. Reply with papers or milestones you think belong here; this can grow into your project's background/related-work section.
@MatiPinto03 — a short history of NLP/evaluation on regulatory drug labels.
TL;DR: From rule-based label NLP → biomedical transformers → LLMs (capable but hallucination-prone) → today's evaluation science (faithfulness, groundedness, LLM-as-judge). OpenRegEval is an evaluation tool in that last wave.
1. Origins
Drug labels were standardized as SPL (Structured Product Labeling) XML and surfaced via DailyMed/openFDA. Early mining turned labels into structured knowledge — e.g. SIDER side-effect resource (Kuhn, 2016).
2. Key milestones
3. Current state
The active problem is benchmarking LLM outputs against authoritative label text — faithfulness scoring with the supporting snippet shown, human-in-the-loop adjudication. That's exactly OpenRegEval's scorecard model.
4. Open challenges & where it's heading
Reliable ground truth, automated hallucination detection, structural variability across labels/agencies, and auditability acceptable to regulators. Frontier: standardized eval frameworks and regulator-grade traceability.
This is a short orientation review, not exhaustive — years/citations are approximate, so verify primary sources before quoting. Reply with papers or milestones you think belong here; this can grow into your project's background/related-work section.