DataAgent-Bench Starter Kit

English | 中文

Official starter kit for the KDD Cup 2026 DataAgent-Bench challenge. The repository reads tasks from data/public/input/ and writes predictions for downstream evaluation.

---

My Implementation (react branch)

This is a fork of the official starter kit. My custom ReAct agent lives on the react branch and includes the following enhancements over the baseline:

What I changed

• Context preloading: Inject file list, CSV headers, SQLite schemas, JSON structures, and doc previews into the first step so the agent starts with a complete data map instead of blindly exploring.
• Hardened JSON parser: Multi-layer fault-tolerant parser strips ```json fences, extracts bare JSON objects via regex, and rejects multi-object responses — surviving malformed model outputs that broke the baseline.
• Self-Verifier for SQL results: LLM sanity-checks SQL result shapes (empty, wrong column count, row explosion) and regenerates with diagnostics. Conservative triggers — never fires on aggregate queries (COUNT/SUM/AVG without GROUP BY) where 1 row is correct.
• Detailed prompt rules (from zing branch + my additions): column pruning, tied-rows via WHERE value = MIN(value) instead of LIMIT 1, no ROUND for precision, regex-based doc extraction, 7 WRONG/CORRECT examples.
• Reflection + retry loop: On step failure, feed the error back to the model with a reflection prompt — usually recovers within 1-2 retries.
• Token budget guards: Soft warning at 50K chars, hard force-answer at 250K, force answer on last 2 steps, force answer after 5 consecutive errors. Prevents the 30-step loop from burning tokens on stuck states.

Results

• Public demo set: ~32 / 50 (Qwen3.5-35B via SiliconFlow)
• Baseline ReAct: ~16% — root cause was a use_xyma=False CLI bug + weak default model; once fixed, my prompt + context preloading alone took it to ~28%, and thinking mode added the remaining +4.

Files changed (vs. baseline)

File	What
src/data_agent_baseline/agents/react.py	ReAct loop with retry, token budget, force-answer guards
src/data_agent_baseline/agents/prompt.py	System prompt, task context builder, reflection prompts
src/data_agent_baseline/agents/model.py	Qwen3 thinking-mode adapter (reasoning_content extraction)
configs/eval.yaml	Custom eval config
REACT_ANALYSIS.md	Architecture analysis — strengths, weaknesses, ceiling
LEARNINGS.md	Full competition retrospective (Chinese)

Retrospective

See LEARNINGS.md for a full writeup covering: why I dropped the custom rule-classifier architecture in favor of ReAct, thinking-mode randomness, the double-edged sword of self-verification, and which prompt rules actually moved the needle.

---

Overview

Item	Value
Dataset input	data/public/input/
Public demo ground truth	data/public/output/task_<id>/gold.csv
Hidden test data	input/ only, no output/
Entry command	uv run dabench <command> --config PATH
Default run output	artifacts/runs/

Quick Start

1. Install uv by following the official guide:

   • https://docs.astral.sh/uv/getting-started/installation/

2. On macOS and Linux, the standalone installer is:

   curl -LsSf https://astral.sh/uv/install.sh | sh

3. Install project dependencies:

   uv sync

4. Confirm the dataset root is visible:

   uv run dabench status --config configs/react_baseline.example.yaml

5. Run the baseline:

   uv run dabench run-benchmark --config configs/react_baseline.example.yaml

Dataset

The public demo dataset lives under data/public/input/. Each task directory follows this structure:

data/public/input/task_<id>/
├── task.json
└── context/

The corresponding public demo answers live separately under data/public/output/task_<id>/gold.csv. Hidden test sets only include input/, so there is no output/ directory there.

task.json contains:

• task_id
• difficulty
• question

The context/ directory may contain one or more of:

• CSV files
• JSON files
• SQLite / DB files
• Text documents

Configuration

An example config file lives at configs/react_baseline.example.yaml.

dataset:
  root_path: data/public/input

agent:
  model: YOUR_MODEL_NAME
  api_base: YOUR_API_BASE_URL
  api_key: YOUR_API_KEY
  max_steps: 16
  temperature: 0.0

run:
  output_dir: artifacts/runs
  run_id:
  max_workers: 4
  task_timeout_seconds: 600

Config fields:

Field	Meaning
dataset.root_path	Root directory of the public demo input/ dataset. Relative paths are resolved from the project root.
agent.model	Model name.
agent.api_base	OpenAI-compatible API base URL.
agent.api_key	API key, read directly from the config file.
agent.max_steps	Maximum ReAct steps per task.
agent.temperature	Sampling temperature.
run.output_dir	Output directory for run artifacts.
run.run_id	Optional run directory name. Defaults to a UTC timestamp if omitted. Must be a single directory name; existing run directories are rejected.
run.max_workers	Parallel worker count for run-benchmark.
run.task_timeout_seconds	Maximum wall-clock time per task. Set to 0 or a negative value to disable the task-level timeout.

CLI

uv run dabench <command> --config PATH [options]

Command	Purpose	Example
status	Show project paths, config path, dataset root, and public task counts.	uv run dabench status --config configs/react_baseline.example.yaml
inspect-task	Show task metadata and list accessible files under context/.	uv run dabench inspect-task task_1 --config configs/react_baseline.local.yaml
run-task	Run the baseline on one task and write outputs.	uv run dabench run-task task_1 --config configs/react_baseline.local.yaml
run-benchmark	Run the baseline across the public dataset.	uv run dabench run-benchmark --config configs/react_baseline.local.yaml

run-benchmark also supports --limit N to cap the number of tasks.

Tools

The baseline exposes these tools to the model:

Tool	Purpose	Inputs
list_context	List files and directories under context/.	max_depth
read_csv	Read a CSV preview.	path, max_rows
read_json	Read a JSON preview.	path, max_chars
read_doc	Read a text document preview.	path, max_chars
inspect_sqlite_schema	Inspect tables in a SQLite / DB file.	path
execute_context_sql	Execute read-only SQL against a SQLite / DB file in context/.	path, sql, limit
execute_python	Execute arbitrary Python code inside the task context/ directory.	code
answer	Submit the final answer table and terminate the task.	columns, rows

All file paths passed to tools must be relative to the task context/ directory.

Outputs

Each successful task run may produce:

• trace.json
• prediction.csv

Per-task outputs are written to:

artifacts/runs/<run_id>/<task_id>/
├── trace.json
└── prediction.csv

Benchmark runs also write:

artifacts/runs/<run_id>/summary.json

Contact

• Open issues: https://github.com/HKUSTDial/kddcup2026-data-agents-starter-kit/issues
• Official website: https://dataagent.top
• Discord: https://discord.com/invite/7eFwJQN3Fx
• WeChat official account: 数据智能与分析实验室 DIAL

Official Website

Discord

WeChat Official Account

Main Modules

Module	Responsibility
src/data_agent_baseline/benchmark/dataset.py	Public dataset loader
src/data_agent_baseline/tools/filesystem.py	list_context, read_csv, read_json, read_doc
src/data_agent_baseline/tools/python_exec.py	execute_python
src/data_agent_baseline/tools/sqlite.py	inspect_sqlite_schema, execute_context_sql
src/data_agent_baseline/tools/registry.py	Tool registration and terminal answer
src/data_agent_baseline/agents/prompt.py	System prompt, task prompt, observation prompt
src/data_agent_baseline/agents/react.py	ReAct runtime with JSON action protocol
src/data_agent_baseline/run/runner.py	Single-task and benchmark execution