TaskDistill

Turn expensive teacher behavior into a small, measurable task model.

TaskDistill is a minimal end-to-end harness for task-specific LLM distillation:

seed cases → teacher examples → filters → locked split → student → eval → report

The orchestration path is one readable Python file and has zero required third-party dependencies. Start with the built-in student to verify the data and evaluation loop. When the task is real, swap in the optional Hugging Face/LoRA trainer or your own command.

Complete GitHub bundle

This repository is distributed as one upload-ready archive. After extracting it, upload the contents of the taskdistill folder to GitHub. Everything required for the first release is already included:

• complete source code, tests, examples, and GitHub Actions;
• the prebuilt wheel at dist/taskdistill-0.1.0-py3-none-any.whl;
• wheel checksums at dist/SHA256SUMS.txt;
• all Markdown documentation under docs/;
• checked-in demo and comparison previews under docs/assets/.

Install directly from the bundled wheel:

python -m pip install dist/taskdistill-0.1.0-py3-none-any.whl
taskdistill --help
taskdistill demo

Included previews

Artifact	Purpose
Demo report	Standalone evaluation dashboard for the offline demo.
Comparison report	Teacher-versus-student quality, latency, token, and cost comparison.
Demo scorecard	Shareable evaluation result card.
Comparison scorecard	Shareable teacher-versus-student result card.

GitHub displays the SVG scorecards directly. Download the repository or open the raw file to view the standalone HTML reports.

Why

General models are impressive and expensive. Many production jobs are narrow:

• route a customs issue to the right team;
• extract fields from one document family;
• classify support tickets;
• normalize product descriptions;
• generate one constrained business response;
• call one tool with a fixed schema.

For these jobs, the useful question is not “which model feels smartest?” It is:

Can a cheaper local student preserve enough of the teacher’s quality under a locked evaluation?

TaskDistill keeps that loop explicit and benchmarkable.

Five-minute start

Requires Python 3.11 or newer.

git clone https://github.com/YOUR_USERNAME/taskdistill.git
cd taskdistill
python taskdistill.py demo --out demo_run

On Windows PowerShell:

python .\taskdistill.py demo --out .\demo_run
start .\demo_run\runs\latest\report.html

The offline demo uses no API key and no network. It:

1. synthesizes 30 examples for a customs-issue router;
2. validates and deduplicates them;
3. makes grouped train/validation/test splits;
4. runs 36 student-recipe experiments;
5. selects the best recipe on validation data;
6. evaluates once on the held-out test set;
7. writes JSON, HTML, CSV, and SVG artifacts.

Typical output:

[1/6] synthesize   30 examples
[2/6] filter       30 kept
[3/6] split        18 train / 6 val / 6 test
[4/6] research     36 experiments
[5/6] evaluate     66.7% exact match
[6/6] report       runs/latest/report.html

The built-in student is intentionally simple. Its job is to prove that your data, split, metric, and reporting path work before you spend GPU time.

Install the CLI

Use the prebuilt wheel bundled in this repository:

python -m pip install dist/taskdistill-0.1.0-py3-none-any.whl
taskdistill --help
taskdistill demo

For editable development from source:

python -m pip install -e .
taskdistill --help

For neural training:

python -m pip install -e ".[train]"

Build a project

taskdistill init my-task
cd my-task

Edit two files:

taskdistill.toml   # task, provider, budget, split, metric
data/seeds.jsonl   # representative seed cases

Then run the full loop:

taskdistill run --config taskdistill.toml --fail-fast

Minimal configuration

[project]
name = "support-router"
task = "Classify a support message as billing, bug, account, or feature. Output only the label."
run_dir = "runs/latest"
seed = 1337

[data]
seeds = "data/seeds.jsonl"
raw = "runs/latest/raw.jsonl"
clean = "runs/latest/clean.jsonl"
splits = "runs/latest/data"

[teacher]
provider = "openai-compatible"
model = "YOUR_TEACHER_MODEL"
base_url = "https://api.openai.com/v1"
api_key_env = "OPENAI_API_KEY"
samples_per_seed = 4
input_price_per_million = 0.0
output_price_per_million = 0.0

[budget]
max_examples = 500
max_teacher_usd = 20.0

[research]
enabled = true
metric = "exact_match"
k_values = [1, 3, 5]
char_weights = [0.0, 0.25, 0.5]
min_scores = [0.0, 0.1]

[eval]
metrics = ["exact_match", "token_f1"]

Seed JSONL:

{"id":"billing-1","input":"I was charged twice","output":"billing","variants":["duplicate payment on my card","two invoices for one order"]}
{"id":"bug-1","input":"the app crashes on launch","output":"bug","variants":["startup immediately closes","blank screen after opening"]}

variants are used only by the deterministic mock teacher. Real teachers receive the seed as context and create new examples.

Teacher adapters

OpenAI-compatible

Works with APIs exposing POST /v1/chat/completions, including many hosted and local servers.

[teacher]
provider = "openai-compatible"
model = "YOUR_MODEL"
base_url = "http://127.0.0.1:11434/v1" # local example
api_key_env = "LOCAL_LLM_KEY"
samples_per_seed = 3

Shell command

The prompt is written to standard input. The command must print one JSON object.

[teacher]
provider = "command"
command = "python my_teacher.py"
timeout_seconds = 120

Mock

Deterministic, offline, and useful for CI:

[teacher]
provider = "mock"

See docs/teachers.md.

Real single-GPU training

train_hf.py is a small causal-LM instruction-tuning script. It masks prompt tokens, trains only on answer tokens, supports LoRA, and can merge the adapter.

python train_hf.py \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --train runs/latest/data/train.jsonl \
  --val runs/latest/data/val.jsonl \
  --out runs/qwen-student \
  --epochs 2 \
  --batch-size 2 \
  --grad-accum 8 \
  --max-length 512 \
  --lora \
  --merge

Evaluate the trained model:

taskdistill eval \
  --backend hf \
  --model runs/qwen-student/merged \
  --data runs/latest/data/test.jsonl \
  --out runs/qwen-student/test.json

Render its report:

taskdistill report \
  --result runs/qwen-student/test.json \
  --title "Qwen 0.5B Customs Router" \
  --html runs/qwen-student/report.html \
  --svg runs/qwen-student/scorecard.svg

Compare the original teacher and student on the same held-out data:

taskdistill benchmark \
  --config taskdistill.toml \
  --data runs/latest/data/test.jsonl \
  --student-backend hf \
  --student-model runs/qwen-student/merged \
  --teacher-backend openai-compatible \
  --student-cost-per-1k 0.31 \
  --out runs/qwen-student/benchmark.json

taskdistill report \
  --result runs/qwen-student/benchmark.json \
  --title "Teacher vs Qwen 0.5B" \
  --html runs/qwen-student/comparison.html \
  --svg runs/qwen-student/comparison.svg

The benchmark reports teacher score, student score, quality retained, p50 latency speedup, provider token usage, estimated teacher cost per 1,000 requests, and the local cost supplied through --student-cost-per-1k.

See docs/training.md.

Commands

Command	Purpose
init	Create a minimal project and configuration.
ingest	Chunk UTF-8 text, Markdown, JSON, CSV, or YAML files.
synthesize	Generate examples through a teacher adapter with budgets and resume.
filter	Validate lengths, remove duplicates, and optionally reject label leakage.
split	Make deterministic grouped train/validation/test splits.
train	Train the built-in baseline or invoke an external training command.
research	Search a fixed recipe grid using validation data only.
eval	Score a memorizer, command, or Hugging Face student.
benchmark	Compare teacher and student quality, latency, tokens, and cost.
predict	Run one or more inputs through a student.
serve	Expose the built-in student on local HTTP.
report	Render standalone HTML and SVG artifacts.
doctor	Check Python, optional packages, configuration, and API-key state.
run	Execute the complete distillation pipeline.
demo	Run the zero-network end-to-end example.

The locked-eval rule

TaskDistill uses three datasets for three distinct purposes:

• train changes model parameters or stores training examples;
• validation chooses recipes, thresholds, checkpoints, and prompts;
• test is touched only for the final reported result.

Generated variants sharing a seed_id remain in the same split. This prevents a paraphrase of the same seed from appearing in both training and test data.

A high score on a contaminated test set is not progress. It is a logging bug.

Output layout

runs/latest/
├── raw.jsonl             # teacher generations + usage metadata
├── clean.jsonl           # validated examples
├── filter.json           # rejection counts
├── data/
│   ├── train.jsonl
│   ├── val.jsonl
│   ├── test.jsonl
│   └── split.json
├── experiments.csv       # fixed-budget recipe search
├── experiments.json      # best experiment summary
├── student.json          # built-in model
├── eval.json             # aggregate and per-example scores
├── report.html           # standalone dashboard
├── scorecard.svg         # shareable result card
└── run.json              # artifact manifest

Local API

Serve the built-in student:

taskdistill serve --model runs/latest/student.json

Request:

curl -X POST http://127.0.0.1:8787/predict \
  -H "Content-Type: application/json" \
  -d '{"input":"customs increased our declared value"}'

Response:

{"prediction":"valuation","meta":{"similarity":0.73}}

Design

The project follows a few hard constraints:

• one obvious entrypoint: taskdistill.py;
• standard-library-only orchestration;
• JSONL and TOML instead of a database;
• fixed teacher budgets;
• deterministic IDs and grouped splits;
• visible training and evaluation loops;
• optional neural complexity, never mandatory complexity;
• artifacts that can be inspected without TaskDistill installed.

The structure is inspired by the public engineering patterns in Karpathy’s nanoGPT, nanochat, llm.c, llama2.c, micrograd, and autoresearch: small surface area, end-to-end execution, explicit budgets, and one metric that decides whether an experiment survived. TaskDistill is an independent project and is not affiliated with Andrej Karpathy.

Documentation

Start with the documentation index. It links the complete documentation set:

• Architecture
• Data format
• Teacher adapters and privacy
• Training students
• Evaluation
• Research loop
• Windows notes
• Publishing to GitHub
• Customs demo
• Generated preview assets

Status

Version 0.1.0 is an alpha release. The core path is tested on Windows and Ubuntu in GitHub Actions. The optional neural trainer is syntax-checked in CI but requires a model download and appropriate hardware for a real run.

License

MIT. See LICENSE.