ObliQA-MultiPassage

This repository builds on the ObliQA questions and answers generated by the RegNLP team:

Source: https://github.com/RegNLP/ObliQADataset

From that original ObliQA dataset, we keep only the questions that are linked to multiple passages, and save them here as:

ObliQA-MultiPassage/  
└── ObliQA_MultiPassage.json

Table of Contents

1. Overview
2. Data
3. Installation & Prerequisites
4. Validation Pipeline
   • Configuration
   • Running the Validator
5. Statistics & Splitting
6. Directory Structure
7. Contributing
8. License

Overview

ObliQA-MultiPassage is a multi-passage extension of the original ObliQA dataset. It supports:

• Passage-level validation (Directly vs. Indirectly vs. Not Connected) via GPT-4
• Automatic filtering to keep only valid multi-passage questions
• Dataset statistics (number of passages per question)
• Train/Validation/Test splits (70/15/15 by default)

Data

1. Original ObliQA
   All questions and single-passage answers were generated by the RegNLP/ObliQADataset pipeline.

2. Multi-Passage Extension
   We filter for questions that reference at least two passages and include at least one “Directly Connected” passage. The result is saved as:

ObliQA-MultiPassage/ObliQA_MultiPassage.json

Installation & Prerequisites

1. Clone or download this folder into your Google Drive (for Colab) or local machine.
2. Install Python 3.8+ and the following packages:

pip install openai pandas

Obtain an OpenAI API key and set it in your environment:

export OPENAI_API_KEY="sk-…"

Validation Pipeline

This step labels each (question, passage) pair as Directly Connected, Indirectly Connected, or Not Connected using GPT-4.

Configuration

Edit the top of validate.py (or the notebook cells) to set:

INPUT_JSON          = "ObliQA_MultiPassage.json"
OUTPUT_JSONL        = "ObliQA_Validated_MultiPassage.jsonl"
OUTPUT_JSON         = "ObliQA_Validated_MultiPassage.json"
CACHE_JSON          = "ObliQA_Validation_Cache.json"
BATCH_PROGRESS_JSON = "Batch_Progress.json"
MODEL               = "gpt-4.1-2025-04-14"
FLUSH_EVERY         = 5
SLEEP_RANGE         = (1, 3)  # seconds between requests
SAMPLE_SIZE         = None   # for testing, set to an integer

Running the Validator python validate.py

• It will skip any (QuestionID, PassageID) already in the cache.

• Progress is written to Batch_Progress.json.

• Final validated data appears in ObliQA_Validated_MultiPassage.json.

Statistics & Splitting

After validation, two helper scripts/notebook cells provide:

1. Dataset statistics

   python stats.py

   – Prints distribution of passages per question, total questions, total passages.

2. Train/Val/Test split

   python split.py

   – By default, splits 70% train / 15% val / 15% test.
   – Outputs:

   • ObliQA_MultiPassage_train.json

   • ObliQA_MultiPassage_val.json

   • ObliQA_MultiPassage_test.json

Below is the distribution of questions by number of associated passages in each split (with totals):

Split	#Questions	2 Passages	3 Passages	4 Passages	5 Passages	6 Passages
Train	2083	1559	382	90	34	18
Validation	446	322	93	16	9	6
Test	447	326	88	21	11	1
Total	2976	2207	563	127	54	25

Directory Structure

`ObliQA-MultiPassage/
├── ObliQA_MultiPassage.json              # filtered multi-passage questions
├── validate.py (or notebook)             # runs the GPT-4 validation loop ├── ObliQA_Validation_Cache.json          # cached GPT responses
├── ObliQA_Validated_MultiPassage.jsonl   # line-delimited validated output
├── ObliQA_Validated_MultiPassage.json    # flushed JSON  version ├── Batch_Progress.json                   # progress tracker
├── stats.py (or notebook cell)           # prints dataset stats
├── split.py (or notebook cell)           # generates train/val/test splits
├── ObliQA_MultiPassage_{train,val,test}.json
└── README.md`

---

Citing

@inproceedings{gokhan-briscoe-2025-grounded,
    title = "Grounded Answers from Multi-Passage Regulations: Learning-to-Rank for Regulatory {RAG}",
    author = "Gokhan, Tuba  and
      Briscoe, Ted",
    editor = "Aletras, Nikolaos  and
      Chalkidis, Ilias  and
      Barrett, Leslie  and
      Goanț{\u{a}}, C{\u{a}}t{\u{a}}lina  and
      Preoțiuc-Pietro, Daniel  and
      Spanakis, Gerasimos",
    booktitle = "Proceedings of the Natural Legal Language Processing Workshop 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.nllp-1.10/",
    doi = "10.18653/v1/2025.nllp-1.10",
    pages = "135--146",
    ISBN = "979-8-89176-338-8",
    abstract = "Regulatory compliance questions often require aggregating evidence from multiple, interrelated sections of long, complex documents. To support question-answering (QA) in this setting, we introduce \textbf{ObliQA-MP}, a dataset for multi-passage regulatory QA, extending the earlier ObliQA benchmark (CITATION), and improve evidence quality with an LLM{--}based validation step that filters out {\textasciitilde}20{\%} of passages missed by prior natural language inference (NLI) based filtering. Our benchmarks show a notable performance drop from single- to multi-passage retrieval, underscoring the challenges of semantic overlap and structural complexity in regulatory texts. To address this, we propose a \textbf{feature-based learning-to-rank (LTR)} framework that integrates lexical, semantic, and graph-derived information, achieving consistent gains over dense and hybrid baselines. We further add a lightweight score-based filter to trim noisy tails and an obligation-centric prompting technique. On ObliQA-MP, LTR improves retrieval (Recall@10/MAP@10/nDCG@10) over dense, hybrid, and fusion baselines. Our generation approach, based on domain-specific filtering plus prompting, achieves strong scores using the RePAS metric (CITATION) on ObliQA-MP, producing faithful, citation-grounded answers. Together, \textbf{ObliQA-MP} and our validation and RAG systems offer a stronger benchmark and a practical recipe for grounded, citation-controlled QA in regulatory domains."
}

Contributing

Feel free to submit issues or pull requests. If you extend the validation logic or add new splits, please update the corresponding scripts and README.