[examples] feat: add DPPO teacher configs for pickscore/ocr/geneval

[Jayce-Ping]

Summary

• Adds three single-reward DPPO teacher training configs under examples/dppo/lora/sd3_5/, migrated from the private flow_dppo/ teacher configs:
  • pickscore_teacher.yaml — PickScore reward (bs=16, async train / bs=32, sync eval); ckpt Jayce-Ping/Pickscore-Teacher
  • ocr_teacher.yaml — OCR reward (bs=64, bfloat16, sync); ckpt Jayce-Ping/OCR-Teacher
  • geneval_teacher.yaml — GenEval reward (bs=64, float32, sync); ckpt Jayce-Ping/GenEval-Teacher
• Each file's first line is a # Checkpoint: <url> comment pointing to the published HuggingFace teacher LoRA, so the config is self-describing for downstream OPD distillation (examples/opd/lora/sd3_5/geneval_pickscore_ocr.yaml teachers[].path).
• Algorithm mapping from private -> current framework:
  • trainer_type: grpo + mask_type: kl_adv -> trainer_type: dppo
  • kl_type: v-based -> kl_type: v-based + kl_mask_type: v-based (mask is now decoupled from the ref-KL space and must be set explicitly)
  • add_kl_coefficient / clip_range -> dropped (DPPO path has no PPO ratio-clip; kl_beta=0 disables the ref term)
• Shared DPPO hyperparams: adv_clip_range=10.0, advantage_aggregation='sum', kl_mask_threshold=1e-6, kl_beta=0, group_size=18, per_device_batch_size=8, max_epochs=1000, CPS SDE (noise_level=0.8, num_sde_steps=3, sde_steps=[0,1,2,3,4]), SD3.5-Medium LoRA r=32 / a=64.
• All three adopt the new data.datasets[] schema with force_reprocess: false (overrides private geneval.yaml's true).
• Comment style follows geneval2_single.yaml (per-line inline comments); DPPO hyperparams are filled from the teacher values, not copied from the geneval2 example (which uses kl_mask_type: x-based and is not equivalent).

Test plan

• All three YAMLs load via Arguments.load_from_yaml with no errors.
• Verified parsed DPPO fields match the plan:
  • trainer_type == 'dppo'
  • kl_type == 'v-based', kl_mask_type == 'v-based'
  • kl_mask_threshold == 1e-6, kl_beta == 0
  • adv_clip_range == (-10.0, 10.0), advantage_aggregation == 'sum'
  • group_size == 18, per_device_batch_size == 8, max_epochs == 1000
• Verified per-file identity: run_name, datasets[0].name, force_reprocess == False, and reward (name/reward_model/batch_size/dtype/async_reward) all match the spec.
• (Manual) Smoke-train one config to confirm the registry resolves PickScore / OCR / GenEval and the CPS SDE scheduler fires — not run here, no GPU available locally.

[Copilot]

Pull request overview

Adds DPPO “teacher” training example configs for SD3.5-Medium LoRA, making three single-reward teacher setups (PickScore / OCR / GenEval) available under examples/ with self-describing checkpoint links for downstream distillation workflows.

Changes:

• Added pickscore_teacher.yaml DPPO teacher config with async PickScore reward during training and sync eval reward settings.
• Added ocr_teacher.yaml DPPO teacher config with OCR reward settings (bf16 reward inference, sync).
• Added geneval_teacher.yaml DPPO teacher config with GenEval reward settings (float32 reward inference, sync) and short GenEval behavior notes.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
examples/dppo/lora/sd3_5/pickscore_teacher.yaml	New DPPO teacher example config for PickScore (async train reward, sync eval reward).
examples/dppo/lora/sd3_5/ocr_teacher.yaml	New DPPO teacher example config for OCR reward (sync).
examples/dppo/lora/sd3_5/geneval_teacher.yaml	New DPPO teacher example config for GenEval reward (float32, sync).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.