From 869d8cdcc80f67f037d44d51ff559647a7057dd8 Mon Sep 17 00:00:00 2001 From: bruceszchen Date: Fri, 31 Jul 2026 19:57:40 +0800 Subject: [PATCH 1/4] docs(datasets): document Daily-Omni and Video-R1-260k prep for Qwen3-Omni RL The two Qwen3-Omni GSPO recipes ship conversion scripts but no instructions, so the upstream download, the intermediate format each converter expects, and the pipeline constraints behind every conversion choice were undocumented. Add a README per dataset covering the source and download commands, the cooking steps, and why each output field is shaped that way: MultimodalRLDataSource only accepts (image, condition), (video, condition) and (video, prompt), so audio reaches Qwen3-Omni through use_audio_in_video rather than an audio media ref, and MCExactMatchRewardScorer returns 0.0 instead of raising on a malformed metadata.answer. Both also recommend pre-filtering prompts whose whole group scores identically, since group-normalized advantages are exactly 0 there and UniRL has no DAPO-style dynamic sampling to skip them at runtime. Note that .gitignore's datasets/* rule covers these files, so they need git add -f, as did the already-tracked converters and image_edit README. Co-authored-by: Cursor --- datasets/daily_omni_av/README.md | 260 +++++++++++++++++++++++++++++++ datasets/video_r1_260k/README.md | 230 +++++++++++++++++++++++++++ 2 files changed, 490 insertions(+) create mode 100644 datasets/daily_omni_av/README.md create mode 100644 datasets/video_r1_260k/README.md diff --git a/datasets/daily_omni_av/README.md b/datasets/daily_omni_av/README.md new file mode 100644 index 00000000..63f37260 --- /dev/null +++ b/datasets/daily_omni_av/README.md @@ -0,0 +1,260 @@ +# Daily-Omni Audio/Video MCQA RL Dataset + +Audio-visual multiple-choice QA prompts for RL training of Qwen3-Omni Thinker with +`use_audio_in_video: true`. Each sample is one video (whose embedded audio track carries +part of the answer) plus a 4-way A/B/C/D question. + +Used by: + +- `examples/ar/qwen3_omni_audio_video_gspo_lora_vllm_omni_1x8.yaml` +- `examples/ar/qwen3_omni_audio_video_gspo_lora_vllm_omni_1x4.yaml` + +## Source + +- **Daily-Omni**: [liarliar/Daily-Omni](https://huggingface.co/datasets/liarliar/Daily-Omni) + - Code: [Lliar-liar/Daily-Omni](https://github.com/Lliar-liar/Daily-Omni) + - Paper: [arXiv:2505.17862](https://arxiv.org/abs/2505.17862) — *Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalities* + - License: **CC BY-NC-SA 4.0** (non-commercial, share-alike) +- 684 YouTube videos (375 × 30 s, 309 × 60 s), 1,197 QA pairs, all 4-way multiple choice. +- QA types: Event Sequence (306), AV Event Alignment (238), Context understanding (193), + Reasoning (175), Inference (154), Comparative (131). +- Every video ships as an MP4 with an **H.264 video track and an AAC audio track**, plus a + pre-extracted `.wav` alongside it. + +> **Daily-Omni is published as an evaluation benchmark, not a training set.** There is no +> official train/val split. The HuggingFace dataset viewer shows a split named `train` with +> 1,197 rows — that is just HF's default name for a single bare `qa.json`, not a training +> partition. If you train on part of it and evaluate on the rest, your `eval/acc` is **not** +> the published Daily-Omni benchmark number, and any leaderboard comparison is contaminated. +> Treat this recipe as a small-scale audio-in-video RL smoke test, and carve the split +> yourself (see below). + +## Download + +```bash +hf download liarliar/Daily-Omni --repo-type dataset --local-dir /path/to/Daily-Omni + +# Videos.tar is an uncompressed tar — do NOT pass -z. +tar -xf /path/to/Daily-Omni/Videos.tar -C /path/to/Daily-Omni +``` + +Total ~3.9 GB compressed; budget ~8 GB while the tar and the extracted tree coexist. +After extraction: + +``` +/path/to/Daily-Omni/ +├── qa.json +└── Videos//_video.mp4 + /_audio.wav +``` + +`qa.json` rows use capitalized keys (and note the misspelled `Explaination`): + +```json +{ + "Question": "What visual elements were displayed immediately after ...?", + "Choice": ["A. ...", "B. ...", "C. ...", "D. ..."], + "Answer": "B", + "video_id": "Ec_lQgZ9wlg", + "Type": "Event Sequence", + "video_duration": "30s" +} +``` + +`content_parent_category` / `content_fine_category` (960 rows) and `Explaination` (235 rows) +are optional — parse defensively. + +## Cook + +Two steps. `convert_daily_omni_dataset_format_to_unirl.py` consumes a **verl/EasyR1-style +JSONL**, not the raw `qa.json`, so you first flatten `qa.json` into that intermediate form +and split it. + +### Step 1 — `qa.json` → verl-style JSONL + +The converter reads four things per row: the user text (`prompt[].content[].text`), the video +path (`videos[0].video`, falling back to a `type: "video"` content part), the gold letter +(`reward_model.ground_truth`), and `extra_info.{video_id,qa_type}`. + +```python +import json, os, random + +ROOT = "/path/to/Daily-Omni" +INSTRUCTION = ( + "Watch the video and listen to its audio, then answer the multiple-choice question.\n" + "Reason step by step, then end your reply with the exact phrase: The answer is [X]" +) + +rows = [] +for i, qa in enumerate(json.load(open(f"{ROOT}/qa.json", encoding="utf-8"))): + vid = qa["video_id"] + path = os.path.join(ROOT, "Videos", vid, f"{vid}_video.mp4") + text = "\n".join([qa["Question"], *qa["Choice"], INSTRUCTION]) + rows.append({ + "prompt": [{"role": "user", "content": [ + {"type": "video", "video": path}, + {"type": "text", "text": text}, + ]}], + "videos": [{"video": path}], + "reward_model": {"ground_truth": qa["Answer"]}, + "extra_info": {"video_id": vid, "qa_type": qa.get("Type")}, + }) + +# Split by video_id so the same clip never lands in both splits. +by_video = {} +for r in rows: + by_video.setdefault(r["extra_info"]["video_id"], []).append(r) +vids = sorted(by_video) +random.Random(42).shuffle(vids) +val_vids = set(vids[: max(1, len(vids) // 10)]) + +for name, keep in (("train", lambda v: v not in val_vids), ("val", lambda v: v in val_vids)): + with open(f"{ROOT}/daily_omni_av_{name}.jsonl", "w", encoding="utf-8") as f: + for v in vids: + if keep(v): + for r in by_video[v]: + f.write(json.dumps(r, ensure_ascii=False) + "\n") +``` + +Two details that matter: + +- **Split by `video_id`, not by row.** Several QA pairs share one clip; a naive row-level + split leaks the same video into train and val and inflates `eval/acc`. +- **The `The answer is [X]` instruction is required.** Both recipes score with + `MCExactMatchSpec(require_answer_phrase: true)`, which only accepts a phrase matching + `(answer|option)\s*(is|:)\s*[\(\[]?([A-D])` or an `X` tag. A reply ending + in a bare `B` scores **0.0**. Daily-Omni's own `Question` field carries no format + instruction, and the converter copies the user text verbatim, so if you skip this the + entire run sits at reward 0. + +### Step 2 — verl-style JSONL → UniRL JSONL + +```bash +python datasets/daily_omni_av/convert_daily_omni_dataset_format_to_unirl.py \ + --train-input /path/to/Daily-Omni/daily_omni_av_train.jsonl \ + --val-input /path/to/Daily-Omni/daily_omni_av_val.jsonl \ + --out-dir datasets/daily_omni_av +``` + +Rows whose MP4 is missing on disk are dropped; pass `--keep-missing` to emit them anyway +(useful for a dry run before the tar finishes extracting). An unparseable ground truth +aborts with the offending `path:line`. + +## Format + +Each output line: + +```json +{ + "prompt": "Question text\nA. ...\nB. ...\nC. ...\nD. ...\nReason step by step, then end your reply with the exact phrase: The answer is [X]", + "prompt_id": "daily_omni_av:train:000042:Ec_lQgZ9wlg", + "media_refs": [{"modality": "video", "role": "prompt", "uri": "/abs/path/Ec_lQgZ9wlg_video.mp4"}], + "metadata": {"answer": "B", "video_id": "Ec_lQgZ9wlg", "qa_type": "Event Sequence"} +} +``` + +## Why cook it this way + +- **One video ref, no audio ref.** `MultimodalRLDataSource` only accepts the pairs + `(image, condition)`, `(video, condition)` and `(video, prompt)`; anything else — including + **any `modality: "audio"` entry** — raises `NotImplementedError` at collate time. Audio + reaches Qwen3-Omni through the MP4's own AAC track: the recipes set + `use_audio_in_video: true` on the bundle, the pipeline and the vLLM-Omni engine, and the + processor demuxes it. So the shipped `_audio.wav` files are **not** referenced. Adding them + as a second media ref would crash the run. +- **`role: "prompt"`, not `"condition"`.** `(video, condition)` decodes the clip into a frame + tensor for diffusion V2V. `(video, prompt)` passes the URI through to the Qwen3-Omni + conversation builder, which is what an AR prompt video needs. A batch may not mix the two. +- **At most one video ref per prompt**, or collate raises `ValueError`. +- **Absolute URIs.** The converter calls `os.path.abspath(os.path.expanduser(...))`. Relative + URIs are resolved against the directory holding the JSONL, so absolute paths keep the file + relocatable. +- **`metadata.answer` is a single uppercase letter.** `MCExactMatchRewardScorer` reads + `metadata["answer"]` and nothing else, and the converter unwraps `[B]` → `B` and validates + membership in `{A,B,C,D}` up front. This matters because the scorer **fails silently**: a + missing or malformed `answer` returns 0.0 rather than raising, which looks identical to a + model that never gets anything right. +- **Unique `prompt_id`.** It becomes the root `sample_id` (`prompt:{id}:sample:0`) and is what + GSPO groups siblings by; duplicates inside one batch raise `ValueError`. The + `{split}:{index}:{video_id}` form stays unique even when several questions share a clip. + +## Usage + +```yaml +data_source: + _target_: unirl.data.data_source.MultimodalRLDataSource + args: + run: + data_path: datasets/daily_omni_av/train.jsonl + eval_data_path: datasets/daily_omni_av/val.jsonl + seed: 42 + algorithm: + prompts_per_rollout: 8 # must equal batch_size +``` + +```bash +QWEN3_OMNI_PATH=/path/to/Qwen3-Omni-30B-A3B-Instruct \ +DATA_PATH=datasets/daily_omni_av/train.jsonl \ +EVAL_DATA_PATH=datasets/daily_omni_av/val.jsonl \ +ENTRY=train_ar bash examples/run_experiment_single_node.sh \ + ar/qwen3_omni_audio_video_gspo_lora_vllm_omni_1x8 +``` + +## Recommended: pre-filter all-zero-reward groups + +GRPO/GSPO advantages are group-normalized (`Part.compute_advantages`, `scope="group"`): + +``` +adv_i = (r_i - mean(r_group)) / (std(r_group) + 1e-8) +``` + +When every sample in a group scores the same, `r_i - mean = 0` and the advantage is +**exactly 0** — with or without `normalize_adv_by_std`. That group consumes a full rollout +(here 8 samples × a 30–60 s video through the audio+vision towers, the most expensive thing +in the loop) and contributes no gradient. Two cases produce it: + +- **all-zero groups** — the model never gets the question right, or never emits the + `The answer is [X]` phrase; +- **all-one groups** — the question is already saturated. + +UniRL has no DAPO-style dynamic sampling: nothing resamples or skips these at runtime. It +only *reports* them, as `rollout/zero_std_group_ratio` and `rollout/zero_std_group_count` in +W&B. Watch those first; if the ratio is high, filter offline. + +The procedure: roll out K samples per prompt with **the exact model you are about to train** +(same checkpoint/adapter, same prompt text, same `temperature`/`top_p`/`max_new_tokens` as +the `sampling:` block), score them with the same `MCExactMatchSpec` settings the recipe uses, +and drop prompts whose K rewards are all identical. + +```python +import json + +K_LO, K_HI = 1, 7 # keep prompts with 1..7 correct out of K=8 +keep = {pid for pid, rs in json.load(open("passrate.json")).items() if K_LO <= sum(rs) <= K_HI} + +with open("train.jsonl", encoding="utf-8") as src, \ + open("train.filtered.jsonl", "w", encoding="utf-8") as dst: + for line in src: + if json.loads(line)["prompt_id"] in keep: + dst.write(line) +``` + +Caveats worth respecting on a set this small: + +- Daily-Omni yields only ~1.1k prompts. Aggressive filtering can leave too few rows — + `MultimodalRLDataSource` refuses to start if the dataset is smaller than + `prompts_per_rollout`, and a tiny set means the loader cycles the same prompts every few + rollouts. Prefer dropping only all-zero groups here, and keep the all-one ones. +- The filter is a snapshot of one checkpoint. As the policy improves, previously all-zero + prompts become learnable and previously mixed ones saturate, so re-estimate every few + hundred rollouts rather than filtering once. +- Sanity-check the format first. If pass rates are near zero *everywhere*, the cause is + usually the missing `The answer is [X]` instruction, not difficulty — filtering would + delete the whole dataset instead of fixing the prompt. + +## Notes + +- `video_fps: 2.0` × `video_max_frames: 64` caps a clip at 32 s of sampled frames, so 60 s + videos are subsampled. Frames plus question must fit `max_prompt_length: 16384`; lower + `video_max_pixels` before lowering `video_max_frames` if you overflow. +- Non-commercial license: do not ship models trained on this without checking CC BY-NC-SA 4.0. diff --git a/datasets/video_r1_260k/README.md b/datasets/video_r1_260k/README.md new file mode 100644 index 00000000..238df222 --- /dev/null +++ b/datasets/video_r1_260k/README.md @@ -0,0 +1,230 @@ +# Video-R1-260k Video MCQA RL Dataset + +Video-only multiple-choice QA prompts for RL training of Qwen3-Omni Thinker (no audio +conditioning). Each sample is one video plus a 4-way A/B/C/D question answered in +`X` form. + +Used by: + +- `examples/ar/qwen3_omni_video_r1_gspo_lora_vllm_omni_1x8.yaml` +- `examples/ar/qwen3_omni_video_r1_gspo_lora_vllm_omni_1x4.yaml` + +## Source + +- **Video-R1-data**: [Video-R1/Video-R1-data](https://huggingface.co/datasets/Video-R1/Video-R1-data) + - Code: [tulerfeng/Video-R1](https://github.com/tulerfeng/Video-R1) + - Paper: [arXiv:2503.21776](https://arxiv.org/abs/2503.21776) — *Video-R1: Reinforcing Video Reasoning in MLLMs* (NeurIPS 2025) + - Card license: `apache-2.0` — see the caveat below. + +The repo ships two annotation files. `Video-R1-260k.json` (263,071 rows) is the RL set and the +only one this converter reads; `Video-R1-COT-165k.json` (165,575 rows) is a CoT-annotated +subset for SFT cold start, not a disjoint split. + +`Video-R1-260k.json` mixes images and video: 146,823 image rows and 116,248 video rows, across +five `problem_type`s (multiple choice 168,769; free-form 38,722; numerical 34,354; OCR 15,886; +regression 5,340). Media lives in per-source folders as independent multi-part zips: + +| Folder | Modality | Rows | Zip parts | Size | +| --- | --- | ---: | ---: | ---: | +| LLaVA-Video-178K | video | 82,676 | 38 | 196.9 GB | +| STAR | video | 11,455 | 4 | 16.4 GB | +| CLEVRER | video | 8,220 | 1 | 0.5 GB | +| NeXT-QA | video | 7,549 | 4 | 18.0 GB | +| PerceptionTest | video | 6,348 | 6 | 28.3 GB | +| Knowledge / Math / Chart / Spatial / OCR / General | image | 146,823 | 12 | 48.9 GB | + +> **License caveat.** The `apache-2.0` label realistically covers the Video-R1 team's own +> curation and annotations. The underlying media is aggregated from CLEVRER, STAR, NeXT-QA, +> PerceptionTest, LLaVA-Video-178K and dozens of upstream sets, several of which are +> research-only. Check the sub-sources individually before any commercial use. + +## Download + +The full repo is ~310 GB. Fetch only the video sources you intend to train on — the converter +takes a `--sources` allowlist, and `LLaVA-Video-178K` alone is 197 GB. + +```bash +ROOT=/path/to/Video-R1-data + +# Annotations + the four small/medium video sources (~63 GB). +hf download Video-R1/Video-R1-data --repo-type dataset --local-dir "$ROOT" \ + --include "*.json" "CLEVRER/*" "STAR/*" "NeXT-QA/*" "PerceptionTest/*" +``` + +`--repo-type dataset` is mandatory. Use `--dry-run` first to see the footprint. Prefer +`hf download` over `git clone`: it resumes, and it filters. + +The `*_part*.zip` files are **independent archives**, not spanned volumes, so each is +extracted on its own and a partial download still yields usable (partial) media: + +```bash +for d in CLEVRER STAR NeXT-QA PerceptionTest; do + for z in "$ROOT/$d"/*_part*.zip; do [ -f "$z" ] && unzip -o -q "$z" -d "$ROOT/$d"; done +done +``` + +Extract **in place** — `path` fields in the JSON are relative to `$ROOT` and only resolve +against the original directory layout. Budget ~2× the download size, or delete each zip after +extracting it. + +Source row schema: + +```json +{ + "problem_id": 1, + "problem": "What happens after the green cube collides with the sphere?", + "data_type": "video", + "problem_type": "multiple choice", + "options": ["A) ...", "B) ...", "C) ...", "D) ..."], + "solution": "D", + "path": "./CLEVRER/video_train/video_00042.mp4", + "data_source": "" +} +``` + +`solution` is always `...` regardless of problem type; `options` is `[]` for +non-multiple-choice rows, and option prefixes are inconsistent across sources (`"A) text"` vs +`"A. text"`). + +## Cook + +```bash +python datasets/video_r1_260k/convert_video_r1_260k_to_unirl.py \ + --data-root "$ROOT" \ + --out-dir datasets/video_r1_260k \ + --sources CLEVRER,STAR,NeXT-QA,PerceptionTest \ + --max-total 20000 --val-count 200 +``` + +Useful flags: `--max-per-source` caps each folder (keeps a big source from dominating), +`--seed` fixes the shuffle so the train/val split is reproducible, `--keep-missing` emits rows +whose MP4 is not on disk yet. The script prints kept/missing counts per source; if everything +lands under `missing:`, the zips are not extracted yet. + +## Format + +Each output line: + +```json +{ + "prompt": "What happens after the green cube collides with the sphere?\nA) ...\nB) ...\nC) ...\nD) ...\nFirst reason step by step about which option is correct. Then output the final answer letter (A, B, C, or D) on its own in the exact format:\nX", + "prompt_id": "video_r1_260k:CLEVRER:42", + "media_refs": [{"modality": "video", "role": "prompt", "uri": "/abs/path/video_00042.mp4"}], + "metadata": {"answer": "D"} +} +``` + +## Why cook it this way + +- **Video multiple-choice rows only.** The converter keeps `data_type == "video"` and + `problem_type == "multiple choice"`, dropping ~57% of the file. Both filters are hard + requirements, not preferences: + - `MultimodalRLDataSource` accepts `(image, condition)`, `(video, condition)` and + `(video, prompt)` and raises `NotImplementedError` on anything else. Image rows would need + `(image, condition)`, which is the diffusion I2V path, not an AR prompt image — and a batch + may not mix condition and prompt media anyway. + - `MCExactMatchRewardScorer` only compares A–D letters. Free-form, numerical, OCR and + regression rows would score 0.0 forever under it. +- **`role: "prompt"`, not `"condition"`.** `(video, condition)` decodes the clip into a frame + tensor for diffusion V2V; `(video, prompt)` hands the URI to the Qwen3-Omni conversation + builder, which is what an AR prompt video needs. +- **The `X` instruction is appended to every prompt.** Recipes score + with `require_answer_tag: true`, where *only* a well-formed tag earns 1.0 and everything else + is 0.0. +- **`metadata.answer` is a single uppercase letter**, pulled out of `solution` (preferring the + `…` tag, falling back to the first standalone A–D). `MCExactMatchRewardScorer` + reads `metadata["answer"]` and nothing else, and **returns 0.0 rather than raising** when it + is missing or malformed — a schema mistake here is indistinguishable from a model that is + always wrong, so the converter validates at conversion time instead. +- **Absolute URIs.** `path` is repo-relative (`./CLEVRER/...`); the converter joins it with + `--data-root`. Relative URIs would otherwise be resolved against the JSONL's own directory. +- **Missing files are skipped by default**, so a partially extracted download produces a + smaller but fully valid dataset rather than crashing mid-rollout. +- **Unique `prompt_id`.** It becomes the root `sample_id` (`prompt:{id}:sample:0`) and is what + GSPO groups siblings by; duplicates inside a batch raise `ValueError`. The + `{source}:{problem_id}` form stays unique across folders since `problem_id` is per-file. +- **Deterministic shuffle then split.** Rows are shuffled with `--seed` before the val holdout, + so `val.jsonl` mixes sources instead of being whichever source happened to land last. + +## Usage + +```yaml +data_source: + _target_: unirl.data.data_source.MultimodalRLDataSource + args: + run: + data_path: datasets/video_r1_260k/train.jsonl + eval_data_path: datasets/video_r1_260k/val.jsonl + seed: 42 + algorithm: + prompts_per_rollout: 8 # must equal batch_size +``` + +```bash +QWEN3_OMNI_PATH=/path/to/Qwen3-Omni-30B-A3B-Instruct \ +DATA_PATH=datasets/video_r1_260k/train.jsonl \ +EVAL_DATA_PATH=datasets/video_r1_260k/val.jsonl \ +ENTRY=train_ar bash examples/run_experiment_single_node.sh \ + ar/qwen3_omni_video_r1_gspo_lora_vllm_omni_1x8 +``` + +## Recommended: pre-filter all-zero-reward groups + +GRPO/GSPO advantages are group-normalized (`Part.compute_advantages`, `scope="group"`): + +``` +adv_i = (r_i - mean(r_group)) / (std(r_group) + 1e-8) +``` + +When every sample in a group scores the same, `r_i - mean = 0` and the advantage is +**exactly 0** — with or without `normalize_adv_by_std`. That group still costs a full rollout +(8 samples × up to 64 decoded frames through the vision tower, plus up to 8k generated tokens) +and contributes no gradient. Two cases produce it: + +- **all-zero groups** — the question is beyond the model, or it never emits a well-formed + `X` tag; +- **all-one groups** — the question is saturated and there is nothing left to learn. + +UniRL has no DAPO-style dynamic sampling: nothing resamples or skips these at runtime. It only +*reports* them, as `rollout/zero_std_group_ratio` and `rollout/zero_std_group_count` in W&B. +Watch those first; if the ratio is high, filter offline. + +The procedure: roll out K samples per prompt with **the exact model you are about to train** +(same checkpoint/adapter, same prompt text, same `temperature`/`top_p`/`max_new_tokens` as the +`sampling:` block), score them with the same `MCExactMatchSpec` settings the recipe uses, and +drop prompts whose K rewards are all identical. + +```python +import json + +K_LO, K_HI = 1, 7 # keep prompts with 1..7 correct out of K=8 +keep = {pid for pid, rs in json.load(open("passrate.json")).items() if K_LO <= sum(rs) <= K_HI} + +with open("train.jsonl", encoding="utf-8") as src, \ + open("train.filtered.jsonl", "w", encoding="utf-8") as dst: + for line in src: + if json.loads(line)["prompt_id"] in keep: + dst.write(line) +``` + +This dataset is the right place to be aggressive about it: with 20k+ candidate prompts you can +afford to drop both tails and still have plenty of rows, and the per-prompt cost of a wasted +video rollout is high. Two things to keep in mind: + +- Score with the *same* reward config as training. A prompt looks all-zero under + `require_answer_tag: true` (1x4) while being half-correct under `graded_format_reward: true` + (1x8), since the latter still pays 0.5 for an untagged correct answer. Filtering with the + wrong scorer throws away learnable prompts. +- The filter is a snapshot of one checkpoint. Previously all-zero prompts become learnable as + the policy improves and mixed ones saturate, so re-estimate every few hundred rollouts, or + keep a held-out slice of the discarded hard prompts to re-admit later. + +Cheaper variants when a full K-sample pass is too expensive: filter on a smaller K (K=4 already +separates the tails well), estimate pass rates on a random subsample and drop whole sources +whose accuracy is pinned at 0 or 1, or use `--max-per-source` to rebalance instead of filtering +per prompt. + +## Notes +- `video_fps: 1.0` × `video_max_frames: 64` caps a clip at 64 sampled frames; frames plus + question must fit `max_prompt_length` (12288 for 1x8, 16384 for 1x4). Lower + `video_max_pixels` before lowering `video_max_frames` if you overflow. From e4fe6166a8f928468ab64bf11efd9b4fbfb7a045 Mon Sep 17 00:00:00 2001 From: bruceszchen Date: Fri, 31 Jul 2026 20:57:46 +0800 Subject: [PATCH 2/4] docs(datasets): update the document of Daily-Omni. --- datasets/daily_omni_av/README.md | 92 ++------- ...vert_daily_omni_dataset_format_to_unirl.py | 191 ++++++++++-------- 2 files changed, 128 insertions(+), 155 deletions(-) diff --git a/datasets/daily_omni_av/README.md b/datasets/daily_omni_av/README.md index 63f37260..d6c6e3f5 100644 --- a/datasets/daily_omni_av/README.md +++ b/datasets/daily_omni_av/README.md @@ -66,79 +66,27 @@ are optional — parse defensively. ## Cook -Two steps. `convert_daily_omni_dataset_format_to_unirl.py` consumes a **verl/EasyR1-style -JSONL**, not the raw `qa.json`, so you first flatten `qa.json` into that intermediate form -and split it. - -### Step 1 — `qa.json` → verl-style JSONL - -The converter reads four things per row: the user text (`prompt[].content[].text`), the video -path (`videos[0].video`, falling back to a `type: "video"` content part), the gold letter -(`reward_model.ground_truth`), and `extra_info.{video_id,qa_type}`. - -```python -import json, os, random - -ROOT = "/path/to/Daily-Omni" -INSTRUCTION = ( - "Watch the video and listen to its audio, then answer the multiple-choice question.\n" - "Reason step by step, then end your reply with the exact phrase: The answer is [X]" -) - -rows = [] -for i, qa in enumerate(json.load(open(f"{ROOT}/qa.json", encoding="utf-8"))): - vid = qa["video_id"] - path = os.path.join(ROOT, "Videos", vid, f"{vid}_video.mp4") - text = "\n".join([qa["Question"], *qa["Choice"], INSTRUCTION]) - rows.append({ - "prompt": [{"role": "user", "content": [ - {"type": "video", "video": path}, - {"type": "text", "text": text}, - ]}], - "videos": [{"video": path}], - "reward_model": {"ground_truth": qa["Answer"]}, - "extra_info": {"video_id": vid, "qa_type": qa.get("Type")}, - }) - -# Split by video_id so the same clip never lands in both splits. -by_video = {} -for r in rows: - by_video.setdefault(r["extra_info"]["video_id"], []).append(r) -vids = sorted(by_video) -random.Random(42).shuffle(vids) -val_vids = set(vids[: max(1, len(vids) // 10)]) - -for name, keep in (("train", lambda v: v not in val_vids), ("val", lambda v: v in val_vids)): - with open(f"{ROOT}/daily_omni_av_{name}.jsonl", "w", encoding="utf-8") as f: - for v in vids: - if keep(v): - for r in by_video[v]: - f.write(json.dumps(r, ensure_ascii=False) + "\n") +```bash +python datasets/daily_omni_av/convert_daily_omni_dataset_format_to_unirl.py \ + --qa-json /path/to/Daily-Omni/qa.json \ + --out-dir datasets/daily_omni_av ``` -Two details that matter: +`--videos-root` defaults to `Videos/` next to `qa.json`; point it elsewhere if you unpacked +the tar somewhere else. `--val-ratio` (default `0.1`) and `--seed` (default `42`) control the +holdout. Rows whose MP4 is missing on disk are dropped with a count on stderr — pass +`--keep-missing` to emit them anyway, which is useful for a dry run before the tar finishes +extracting. An unparseable ground truth aborts with the offending `qa.json[index]`. + +Two things the script does that are easy to get wrong by hand: -- **Split by `video_id`, not by row.** Several QA pairs share one clip; a naive row-level - split leaks the same video into train and val and inflates `eval/acc`. -- **The `The answer is [X]` instruction is required.** Both recipes score with +- **It splits by `video_id`, not by row.** Several QA pairs share one clip; a row-level split + leaks the same video into train and val and inflates `eval/acc`. +- **It appends the `The answer is [X]` instruction to every prompt.** Both recipes score with `MCExactMatchSpec(require_answer_phrase: true)`, which only accepts a phrase matching `(answer|option)\s*(is|:)\s*[\(\[]?([A-D])` or an `X` tag. A reply ending - in a bare `B` scores **0.0**. Daily-Omni's own `Question` field carries no format - instruction, and the converter copies the user text verbatim, so if you skip this the - entire run sits at reward 0. - -### Step 2 — verl-style JSONL → UniRL JSONL - -```bash -python datasets/daily_omni_av/convert_daily_omni_dataset_format_to_unirl.py \ - --train-input /path/to/Daily-Omni/daily_omni_av_train.jsonl \ - --val-input /path/to/Daily-Omni/daily_omni_av_val.jsonl \ - --out-dir datasets/daily_omni_av -``` - -Rows whose MP4 is missing on disk are dropped; pass `--keep-missing` to emit them anyway -(useful for a dry run before the tar finishes extracting). An unparseable ground truth -aborts with the offending `path:line`. + in a bare `B` scores **0.0**, and Daily-Omni's own `Question` field carries no format + instruction, so without this line the entire run would sit at reward 0. ## Format @@ -146,7 +94,7 @@ Each output line: ```json { - "prompt": "Question text\nA. ...\nB. ...\nC. ...\nD. ...\nReason step by step, then end your reply with the exact phrase: The answer is [X]", + "prompt": "Question text\nA. ...\nB. ...\nC. ...\nD. ...\nWatch the video and listen to its audio, then answer the multiple-choice question.\nReason step by step, then end your reply with the exact phrase: The answer is [X]", "prompt_id": "daily_omni_av:train:000042:Ec_lQgZ9wlg", "media_refs": [{"modality": "video", "role": "prompt", "uri": "/abs/path/Ec_lQgZ9wlg_video.mp4"}], "metadata": {"answer": "B", "video_id": "Ec_lQgZ9wlg", "qa_type": "Event Sequence"} @@ -166,9 +114,9 @@ Each output line: tensor for diffusion V2V. `(video, prompt)` passes the URI through to the Qwen3-Omni conversation builder, which is what an AR prompt video needs. A batch may not mix the two. - **At most one video ref per prompt**, or collate raises `ValueError`. -- **Absolute URIs.** The converter calls `os.path.abspath(os.path.expanduser(...))`. Relative - URIs are resolved against the directory holding the JSONL, so absolute paths keep the file - relocatable. +- **Absolute URIs.** The converter resolves every clip against `--videos-root` and writes the + absolute path. Relative URIs would instead be resolved against the directory holding the + JSONL, so absolute paths keep the output relocatable. - **`metadata.answer` is a single uppercase letter.** `MCExactMatchRewardScorer` reads `metadata["answer"]` and nothing else, and the converter unwraps `[B]` → `B` and validates membership in `{A,B,C,D}` up front. This matters because the scorer **fails silently**: a diff --git a/datasets/daily_omni_av/convert_daily_omni_dataset_format_to_unirl.py b/datasets/daily_omni_av/convert_daily_omni_dataset_format_to_unirl.py index 32f5afe5..e5c137a4 100755 --- a/datasets/daily_omni_av/convert_daily_omni_dataset_format_to_unirl.py +++ b/datasets/daily_omni_av/convert_daily_omni_dataset_format_to_unirl.py @@ -1,13 +1,25 @@ #!/usr/bin/env python3 -"""Convert Daily-Omni audio/video MCQA data to UniRL JSONL. +"""Build the local daily_omni_av dataset the ``qwen3_omni_audio_video_gspo_*`` recipes train on. -The output contains only a video media reference. Qwen3-Omni extracts the +Reads the official Daily-Omni release — ``qa.json`` plus the ``Videos/`` tree unpacked from +``Videos.tar`` (https://huggingface.co/datasets/liarliar/Daily-Omni) — and writes the jsonl +that ``unirl.data.data_source.MultimodalRLDataSource`` expects:: + + {"prompt": , + "prompt_id": "daily_omni_av:::", + "media_refs": [{"modality": "video", "role": "prompt", "uri": "/_video.mp4"}], + "metadata": {"answer": "", "video_id": , "qa_type": }} + +Only a video media ref is emitted: the loader has no audio role, and Qwen3-Omni extracts the embedded audio track from that same file when ``use_audio_in_video=true``. +Daily-Omni is published as a benchmark with no official split, so the split is carved here by +``video_id`` — several questions share one clip, and splitting per row would leak the same +video into both sides. + Example: python datasets/daily_omni_av/convert_daily_omni_dataset_format_to_unirl.py \ - --train-input /path/to/daily_omni_av_train.jsonl \ - --val-input /path/to/daily_omni_av_val.jsonl \ + --qa-json /path/to/Daily-Omni/qa.json \ --out-dir datasets/daily_omni_av """ @@ -16,44 +28,20 @@ import argparse import json import os -from typing import Any, Dict, Iterable - - -def _user_text(row: Dict[str, Any]) -> str: - messages = row.get("prompt") or [] - user = next((message for message in reversed(messages) if message.get("role") == "user"), None) - if user is None: - raise ValueError("row has no user message") - content = user.get("content", "") - if isinstance(content, str): - return content.strip() - for part in content: - if isinstance(part, dict) and part.get("type") == "text": - return str(part.get("text", "")).strip() - raise ValueError("user message has no text content") - - -def _video_uri(row: Dict[str, Any]) -> str: - videos = row.get("videos") or [] - if videos: - first = videos[0] - uri = first.get("video") if isinstance(first, dict) else first - if uri: - return os.path.abspath(os.path.expanduser(str(uri))) - - messages = row.get("prompt") or [] - for message in reversed(messages): - content = message.get("content", []) - if not isinstance(content, list): - continue - for part in content: - if isinstance(part, dict) and part.get("type") == "video" and part.get("video"): - return os.path.abspath(os.path.expanduser(str(part["video"]))) - raise ValueError("row has no video path") - - -def _answer(row: Dict[str, Any]) -> str: - answer = str((row.get("reward_model") or {}).get("ground_truth", "")).strip().upper() +import random +from typing import Any, Dict, List + +# The recipes score with MCExactMatchSpec(require_answer_phrase=True), which only accepts an +# "answer is X" phrase or an X tag. Daily-Omni's own question text carries no +# format instruction, so without this appended line every rollout would score 0. +ANSWER_INSTRUCTION = ( + "Watch the video and listen to its audio, then answer the multiple-choice question.\n" + "Reason step by step, then end your reply with the exact phrase: The answer is [X]" +) + + +def _answer(raw: Any) -> str: + answer = str(raw).strip().upper() if len(answer) == 3 and answer[0] == "[" and answer[-1] == "]": answer = answer[1] if answer not in {"A", "B", "C", "D"}: @@ -61,66 +49,103 @@ def _answer(row: Dict[str, Any]) -> str: return answer -def _iter_records(path: str, split: str, keep_missing: bool) -> Iterable[Dict[str, Any]]: - with open(path, encoding="utf-8") as source: - output_index = 0 - for source_index, line in enumerate(source): - if not line.strip(): +def _prompt(row: Dict[str, Any]) -> str: + question = str(row["Question"]).strip() + choices = [str(choice).strip() for choice in row["Choice"]] + if not question or not choices: + raise ValueError("row has no question or no choices") + return "\n".join([question, *choices, ANSWER_INSTRUCTION]) + + +def _group_by_video(qa_json: str, videos_root: str, keep_missing: bool) -> Dict[str, List[Dict[str, Any]]]: + with open(qa_json, encoding="utf-8") as source: + rows = json.load(source) + + grouped: Dict[str, List[Dict[str, Any]]] = {} + missing = 0 + for index, row in enumerate(rows): + try: + video_id = str(row["video_id"]) + video = os.path.abspath(os.path.join(videos_root, video_id, f"{video_id}_video.mp4")) + if not keep_missing and not os.path.isfile(video): + missing += 1 continue - row = json.loads(line) - try: - video = _video_uri(row) - if not keep_missing and not os.path.isfile(video): - continue - extra = row.get("extra_info") or {} - video_id = str(extra.get("video_id") or source_index) - yield { - "prompt": _user_text(row), - "prompt_id": f"daily_omni_av:{split}:{output_index:06d}:{video_id}", + grouped.setdefault(video_id, []).append( + { + "prompt": _prompt(row), "media_refs": [{"modality": "video", "role": "prompt", "uri": video}], "metadata": { - "answer": _answer(row), + "answer": _answer(row["Answer"]), "video_id": video_id, - "qa_type": extra.get("qa_type"), + "qa_type": row.get("Type"), }, } - output_index += 1 - except (KeyError, TypeError, ValueError) as exc: - raise ValueError(f"{path}:{source_index + 1}: {exc}") from exc + ) + except (KeyError, TypeError, ValueError) as exc: + raise ValueError(f"{qa_json}[{index}]: {exc}") from exc + + if missing: + print(f"[stats] skipped {missing} rows whose mp4 is not on disk (pass --keep-missing to emit them)") + return grouped + + +def _split(grouped: Dict[str, List[Dict[str, Any]]], val_ratio: float, seed: int) -> Dict[str, List[Dict[str, Any]]]: + video_ids = sorted(grouped) + random.Random(seed).shuffle(video_ids) + + n_val = 0 + if val_ratio > 0 and len(video_ids) > 1: + n_val = max(1, min(int(len(video_ids) * val_ratio), len(video_ids) - 1)) + val_ids = set(video_ids[:n_val]) + + splits: Dict[str, List[Dict[str, Any]]] = {"train": [], "val": []} + for video_id in video_ids: + name = "val" if video_id in val_ids else "train" + for payload in grouped[video_id]: + splits[name].append( + { + "prompt": payload["prompt"], + "prompt_id": f"daily_omni_av:{name}:{len(splits[name]):06d}:{video_id}", + "media_refs": payload["media_refs"], + "metadata": payload["metadata"], + } + ) + return splits -def _write_split(input_path: str, output_path: str, split: str, keep_missing: bool) -> int: - count = 0 +def _write_split(records: List[Dict[str, Any]], output_path: str) -> None: with open(output_path, "w", encoding="utf-8") as output: - for record in _iter_records(input_path, split, keep_missing): + for record in records: output.write(json.dumps(record, ensure_ascii=False) + "\n") - count += 1 - print(f"[write] {output_path}: {count} rows") - return count + print(f"[write] {output_path}: {len(records)} rows") def main() -> None: parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter) - parser.add_argument("--train-input", required=True, help="Daily-Omni training JSONL") - parser.add_argument("--val-input", help="Daily-Omni validation JSONL") + parser.add_argument("--qa-json", required=True, help="official Daily-Omni qa.json") + parser.add_argument("--videos-root", help="unpacked Videos/ tree (default: Videos/ next to qa.json)") parser.add_argument("--out-dir", default="datasets/daily_omni_av") + parser.add_argument("--val-ratio", type=float, default=0.1, help="fraction of videos held out for val.jsonl") + parser.add_argument("--seed", type=int, default=42, help="shuffle seed (deterministic split)") parser.add_argument("--keep-missing", action="store_true", help="keep rows whose video is unavailable locally") args = parser.parse_args() - os.makedirs(args.out_dir, exist_ok=True) - _write_split( - args.train_input, - os.path.join(args.out_dir, "train.jsonl"), - "train", - args.keep_missing, + qa_json = os.path.abspath(os.path.expanduser(args.qa_json)) + videos_root = ( + os.path.abspath(os.path.expanduser(args.videos_root)) + if args.videos_root + else os.path.join(os.path.dirname(qa_json), "Videos") ) - if args.val_input: - _write_split( - args.val_input, - os.path.join(args.out_dir, "val.jsonl"), - "val", - args.keep_missing, - ) + + grouped = _group_by_video(qa_json, videos_root, args.keep_missing) + if not grouped: + raise SystemExit(f"No usable rows. Is {videos_root} unpacked? (tar -xf Videos.tar)") + + splits = _split(grouped, args.val_ratio, args.seed) + os.makedirs(args.out_dir, exist_ok=True) + _write_split(splits["train"], os.path.join(args.out_dir, "train.jsonl")) + if splits["val"]: + _write_split(splits["val"], os.path.join(args.out_dir, "val.jsonl")) if __name__ == "__main__": From 86ccd0e9dcc062684f34f8f69d3af2e2c18c5c04 Mon Sep 17 00:00:00 2001 From: bruceszchen Date: Fri, 31 Jul 2026 21:08:40 +0800 Subject: [PATCH 3/4] docs(datasets): fix typo of the document of Daily-Omni. --- .../daily_omni_av/convert_daily_omni_dataset_format_to_unirl.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/datasets/daily_omni_av/convert_daily_omni_dataset_format_to_unirl.py b/datasets/daily_omni_av/convert_daily_omni_dataset_format_to_unirl.py index e5c137a4..cdd4cfa6 100755 --- a/datasets/daily_omni_av/convert_daily_omni_dataset_format_to_unirl.py +++ b/datasets/daily_omni_av/convert_daily_omni_dataset_format_to_unirl.py @@ -1,5 +1,5 @@ #!/usr/bin/env python3 -"""Build the local daily_omni_av dataset the ``qwen3_omni_audio_video_gspo_*`` recipes train on. +"""Convert Daily-Omni audio/video MCQA data to UniRL JSONL. Reads the official Daily-Omni release — ``qa.json`` plus the ``Videos/`` tree unpacked from ``Videos.tar`` (https://huggingface.co/datasets/liarliar/Daily-Omni) — and writes the jsonl From 73d4f7570dee6d4d43837b95799ae010ad840271 Mon Sep 17 00:00:00 2001 From: Haonan Wang Date: Fri, 31 Jul 2026 21:11:00 +0800 Subject: [PATCH 4/4] docs(datasets): streamline Qwen3-Omni data guides --- datasets/daily_omni_av/README.md | 108 ++++-------------- ...vert_daily_omni_dataset_format_to_unirl.py | 5 +- datasets/video_r1_260k/README.md | 69 ++--------- 3 files changed, 29 insertions(+), 153 deletions(-) diff --git a/datasets/daily_omni_av/README.md b/datasets/daily_omni_av/README.md index d6c6e3f5..7b3f5235 100644 --- a/datasets/daily_omni_av/README.md +++ b/datasets/daily_omni_av/README.md @@ -15,19 +15,13 @@ Used by: - Code: [Lliar-liar/Daily-Omni](https://github.com/Lliar-liar/Daily-Omni) - Paper: [arXiv:2505.17862](https://arxiv.org/abs/2505.17862) — *Daily-Omni: Towards Audio-Visual Reasoning with Temporal Alignment across Modalities* - License: **CC BY-NC-SA 4.0** (non-commercial, share-alike) -- 684 YouTube videos (375 × 30 s, 309 × 60 s), 1,197 QA pairs, all 4-way multiple choice. -- QA types: Event Sequence (306), AV Event Alignment (238), Context understanding (193), - Reasoning (175), Inference (154), Comparative (131). -- Every video ships as an MP4 with an **H.264 video track and an AAC audio track**, plus a - pre-extracted `.wav` alongside it. - -> **Daily-Omni is published as an evaluation benchmark, not a training set.** There is no -> official train/val split. The HuggingFace dataset viewer shows a split named `train` with -> 1,197 rows — that is just HF's default name for a single bare `qa.json`, not a training -> partition. If you train on part of it and evaluate on the rest, your `eval/acc` is **not** -> the published Daily-Omni benchmark number, and any leaderboard comparison is contaminated. -> Treat this recipe as a small-scale audio-in-video RL smoke test, and carve the split -> yourself (see below). +- 684 YouTube videos (375 × 30 s, 309 × 60 s) and 1,197 four-way multiple-choice QA pairs. +- Each MP4 contains H.264 video and AAC audio; the release also includes a standalone `.wav`. + +> **Daily-Omni is an evaluation benchmark with no official train/val split.** Hugging Face's +> `train` label is only the default name for the single `qa.json`. Training on a subset makes +> the resulting `eval/acc` incomparable with the published benchmark; treat this recipe as an +> audio-in-video RL smoke test. ## Download @@ -38,7 +32,7 @@ hf download liarliar/Daily-Omni --repo-type dataset --local-dir /path/to/Daily-O tar -xf /path/to/Daily-Omni/Videos.tar -C /path/to/Daily-Omni ``` -Total ~3.9 GB compressed; budget ~8 GB while the tar and the extracted tree coexist. +The download is ~3.9 GB; budget ~8 GB while the tar and extracted tree coexist. After extraction: ``` @@ -48,7 +42,7 @@ After extraction: /_audio.wav ``` -`qa.json` rows use capitalized keys (and note the misspelled `Explaination`): +`qa.json` rows use capitalized keys: ```json { @@ -61,9 +55,6 @@ After extraction: } ``` -`content_parent_category` / `content_fine_category` (960 rows) and `Explaination` (235 rows) -are optional — parse defensively. - ## Cook ```bash @@ -74,19 +65,17 @@ python datasets/daily_omni_av/convert_daily_omni_dataset_format_to_unirl.py \ `--videos-root` defaults to `Videos/` next to `qa.json`; point it elsewhere if you unpacked the tar somewhere else. `--val-ratio` (default `0.1`) and `--seed` (default `42`) control the -holdout. Rows whose MP4 is missing on disk are dropped with a count on stderr — pass +holdout. Rows whose MP4 is missing on disk are dropped and counted — pass `--keep-missing` to emit them anyway, which is useful for a dry run before the tar finishes extracting. An unparseable ground truth aborts with the offending `qa.json[index]`. -Two things the script does that are easy to get wrong by hand: +Two details matter: - **It splits by `video_id`, not by row.** Several QA pairs share one clip; a row-level split leaks the same video into train and val and inflates `eval/acc`. -- **It appends the `The answer is [X]` instruction to every prompt.** Both recipes score with - `MCExactMatchSpec(require_answer_phrase: true)`, which only accepts a phrase matching - `(answer|option)\s*(is|:)\s*[\(\[]?([A-D])` or an `X` tag. A reply ending - in a bare `B` scores **0.0**, and Daily-Omni's own `Question` field carries no format - instruction, so without this line the entire run would sit at reward 0. +- **It appends an explicit answer-format instruction.** Both recipes use + `require_answer_phrase: true`, so a bare `B` scores 0; the reply must contain an + `answer is B` phrase or an `B` tag. ## Format @@ -103,20 +92,15 @@ Each output line: ## Why cook it this way -- **One video ref, no audio ref.** `MultimodalRLDataSource` only accepts the pairs - `(image, condition)`, `(video, condition)` and `(video, prompt)`; anything else — including - **any `modality: "audio"` entry** — raises `NotImplementedError` at collate time. Audio - reaches Qwen3-Omni through the MP4's own AAC track: the recipes set - `use_audio_in_video: true` on the bundle, the pipeline and the vLLM-Omni engine, and the - processor demuxes it. So the shipped `_audio.wav` files are **not** referenced. Adding them - as a second media ref would crash the run. +- **One video ref, no audio ref.** `MultimodalRLDataSource` does not support audio media refs. + Qwen3-Omni instead reads the MP4's AAC track because the recipes set + `use_audio_in_video: true`; the standalone `_audio.wav` is not referenced. - **`role: "prompt"`, not `"condition"`.** `(video, condition)` decodes the clip into a frame tensor for diffusion V2V. `(video, prompt)` passes the URI through to the Qwen3-Omni conversation builder, which is what an AR prompt video needs. A batch may not mix the two. -- **At most one video ref per prompt**, or collate raises `ValueError`. - **Absolute URIs.** The converter resolves every clip against `--videos-root` and writes the - absolute path. Relative URIs would instead be resolved against the directory holding the - JSONL, so absolute paths keep the output relocatable. + absolute path. This avoids dependence on the training process's working directory, but the + output is machine-local; rerun the converter if the video tree moves. - **`metadata.answer` is a single uppercase letter.** `MCExactMatchRewardScorer` reads `metadata["answer"]` and nothing else, and the converter unwraps `[B]` → `B` and validates membership in `{A,B,C,D}` up front. This matters because the scorer **fails silently**: a @@ -137,7 +121,7 @@ data_source: eval_data_path: datasets/daily_omni_av/val.jsonl seed: 42 algorithm: - prompts_per_rollout: 8 # must equal batch_size + prompts_per_rollout: ${batch_size} ``` ```bash @@ -148,58 +132,6 @@ ENTRY=train_ar bash examples/run_experiment_single_node.sh \ ar/qwen3_omni_audio_video_gspo_lora_vllm_omni_1x8 ``` -## Recommended: pre-filter all-zero-reward groups - -GRPO/GSPO advantages are group-normalized (`Part.compute_advantages`, `scope="group"`): - -``` -adv_i = (r_i - mean(r_group)) / (std(r_group) + 1e-8) -``` - -When every sample in a group scores the same, `r_i - mean = 0` and the advantage is -**exactly 0** — with or without `normalize_adv_by_std`. That group consumes a full rollout -(here 8 samples × a 30–60 s video through the audio+vision towers, the most expensive thing -in the loop) and contributes no gradient. Two cases produce it: - -- **all-zero groups** — the model never gets the question right, or never emits the - `The answer is [X]` phrase; -- **all-one groups** — the question is already saturated. - -UniRL has no DAPO-style dynamic sampling: nothing resamples or skips these at runtime. It -only *reports* them, as `rollout/zero_std_group_ratio` and `rollout/zero_std_group_count` in -W&B. Watch those first; if the ratio is high, filter offline. - -The procedure: roll out K samples per prompt with **the exact model you are about to train** -(same checkpoint/adapter, same prompt text, same `temperature`/`top_p`/`max_new_tokens` as -the `sampling:` block), score them with the same `MCExactMatchSpec` settings the recipe uses, -and drop prompts whose K rewards are all identical. - -```python -import json - -K_LO, K_HI = 1, 7 # keep prompts with 1..7 correct out of K=8 -keep = {pid for pid, rs in json.load(open("passrate.json")).items() if K_LO <= sum(rs) <= K_HI} - -with open("train.jsonl", encoding="utf-8") as src, \ - open("train.filtered.jsonl", "w", encoding="utf-8") as dst: - for line in src: - if json.loads(line)["prompt_id"] in keep: - dst.write(line) -``` - -Caveats worth respecting on a set this small: - -- Daily-Omni yields only ~1.1k prompts. Aggressive filtering can leave too few rows — - `MultimodalRLDataSource` refuses to start if the dataset is smaller than - `prompts_per_rollout`, and a tiny set means the loader cycles the same prompts every few - rollouts. Prefer dropping only all-zero groups here, and keep the all-one ones. -- The filter is a snapshot of one checkpoint. As the policy improves, previously all-zero - prompts become learnable and previously mixed ones saturate, so re-estimate every few - hundred rollouts rather than filtering once. -- Sanity-check the format first. If pass rates are near zero *everywhere*, the cause is - usually the missing `The answer is [X]` instruction, not difficulty — filtering would - delete the whole dataset instead of fixing the prompt. - ## Notes - `video_fps: 2.0` × `video_max_frames: 64` caps a clip at 32 s of sampled frames, so 60 s diff --git a/datasets/daily_omni_av/convert_daily_omni_dataset_format_to_unirl.py b/datasets/daily_omni_av/convert_daily_omni_dataset_format_to_unirl.py index cdd4cfa6..8abcafd8 100755 --- a/datasets/daily_omni_av/convert_daily_omni_dataset_format_to_unirl.py +++ b/datasets/daily_omni_av/convert_daily_omni_dataset_format_to_unirl.py @@ -31,9 +31,8 @@ import random from typing import Any, Dict, List -# The recipes score with MCExactMatchSpec(require_answer_phrase=True), which only accepts an -# "answer is X" phrase or an X tag. Daily-Omni's own question text carries no -# format instruction, so without this appended line every rollout would score 0. +# The recipes score with MCExactMatchSpec(require_answer_phrase=True), so append the expected +# format instead of relying on the model to infer it from the answer choices. ANSWER_INSTRUCTION = ( "Watch the video and listen to its audio, then answer the multiple-choice question.\n" "Reason step by step, then end your reply with the exact phrase: The answer is [X]" diff --git a/datasets/video_r1_260k/README.md b/datasets/video_r1_260k/README.md index 238df222..24460367 100644 --- a/datasets/video_r1_260k/README.md +++ b/datasets/video_r1_260k/README.md @@ -48,7 +48,7 @@ ROOT=/path/to/Video-R1-data # Annotations + the four small/medium video sources (~63 GB). hf download Video-R1/Video-R1-data --repo-type dataset --local-dir "$ROOT" \ - --include "*.json" "CLEVRER/*" "STAR/*" "NeXT-QA/*" "PerceptionTest/*" + --include "Video-R1-260k.json" "CLEVRER/*" "STAR/*" "NeXT-QA/*" "PerceptionTest/*" ``` `--repo-type dataset` is mandatory. Use `--dry-run` first to see the footprint. Prefer @@ -128,16 +128,16 @@ Each output line: - **`role: "prompt"`, not `"condition"`.** `(video, condition)` decodes the clip into a frame tensor for diffusion V2V; `(video, prompt)` hands the URI to the Qwen3-Omni conversation builder, which is what an AR prompt video needs. -- **The `X` instruction is appended to every prompt.** Recipes score - with `require_answer_tag: true`, where *only* a well-formed tag earns 1.0 and everything else - is 0.0. +- **The `X` instruction is appended to every prompt.** The 1x4 recipe uses + strict `require_answer_tag: true`; the 1x8 recipe uses `graded_format_reward: true`, which + gives 1.0 for a correct tag and 0.5 for a correct answer in another recognized format. - **`metadata.answer` is a single uppercase letter**, pulled out of `solution` (preferring the `…` tag, falling back to the first standalone A–D). `MCExactMatchRewardScorer` reads `metadata["answer"]` and nothing else, and **returns 0.0 rather than raising** when it is missing or malformed — a schema mistake here is indistinguishable from a model that is always wrong, so the converter validates at conversion time instead. - **Absolute URIs.** `path` is repo-relative (`./CLEVRER/...`); the converter joins it with - `--data-root`. Relative URIs would otherwise be resolved against the JSONL's own directory. + `--data-root` and writes an absolute path. Rerun the converter if the media tree moves. - **Missing files are skipped by default**, so a partially extracted download produces a smaller but fully valid dataset rather than crashing mid-rollout. - **Unique `prompt_id`.** It becomes the root `sample_id` (`prompt:{id}:sample:0`) and is what @@ -157,7 +157,7 @@ data_source: eval_data_path: datasets/video_r1_260k/val.jsonl seed: 42 algorithm: - prompts_per_rollout: 8 # must equal batch_size + prompts_per_rollout: ${batch_size} ``` ```bash @@ -168,63 +168,8 @@ ENTRY=train_ar bash examples/run_experiment_single_node.sh \ ar/qwen3_omni_video_r1_gspo_lora_vllm_omni_1x8 ``` -## Recommended: pre-filter all-zero-reward groups - -GRPO/GSPO advantages are group-normalized (`Part.compute_advantages`, `scope="group"`): - -``` -adv_i = (r_i - mean(r_group)) / (std(r_group) + 1e-8) -``` - -When every sample in a group scores the same, `r_i - mean = 0` and the advantage is -**exactly 0** — with or without `normalize_adv_by_std`. That group still costs a full rollout -(8 samples × up to 64 decoded frames through the vision tower, plus up to 8k generated tokens) -and contributes no gradient. Two cases produce it: - -- **all-zero groups** — the question is beyond the model, or it never emits a well-formed - `X` tag; -- **all-one groups** — the question is saturated and there is nothing left to learn. - -UniRL has no DAPO-style dynamic sampling: nothing resamples or skips these at runtime. It only -*reports* them, as `rollout/zero_std_group_ratio` and `rollout/zero_std_group_count` in W&B. -Watch those first; if the ratio is high, filter offline. - -The procedure: roll out K samples per prompt with **the exact model you are about to train** -(same checkpoint/adapter, same prompt text, same `temperature`/`top_p`/`max_new_tokens` as the -`sampling:` block), score them with the same `MCExactMatchSpec` settings the recipe uses, and -drop prompts whose K rewards are all identical. - -```python -import json - -K_LO, K_HI = 1, 7 # keep prompts with 1..7 correct out of K=8 -keep = {pid for pid, rs in json.load(open("passrate.json")).items() if K_LO <= sum(rs) <= K_HI} - -with open("train.jsonl", encoding="utf-8") as src, \ - open("train.filtered.jsonl", "w", encoding="utf-8") as dst: - for line in src: - if json.loads(line)["prompt_id"] in keep: - dst.write(line) -``` - -This dataset is the right place to be aggressive about it: with 20k+ candidate prompts you can -afford to drop both tails and still have plenty of rows, and the per-prompt cost of a wasted -video rollout is high. Two things to keep in mind: - -- Score with the *same* reward config as training. A prompt looks all-zero under - `require_answer_tag: true` (1x4) while being half-correct under `graded_format_reward: true` - (1x8), since the latter still pays 0.5 for an untagged correct answer. Filtering with the - wrong scorer throws away learnable prompts. -- The filter is a snapshot of one checkpoint. Previously all-zero prompts become learnable as - the policy improves and mixed ones saturate, so re-estimate every few hundred rollouts, or - keep a held-out slice of the discarded hard prompts to re-admit later. - -Cheaper variants when a full K-sample pass is too expensive: filter on a smaller K (K=4 already -separates the tails well), estimate pass rates on a random subsample and drop whole sources -whose accuracy is pinned at 0 or 1, or use `--max-per-source` to rebalance instead of filtering -per prompt. - ## Notes + - `video_fps: 1.0` × `video_max_frames: 64` caps a clip at 64 sampled frames; frames plus question must fit `max_prompt_length` (12288 for 1x8, 16384 for 1x4). Lower `video_max_pixels` before lowering `video_max_frames` if you overflow.