Skip to content

Implement Single-Head Multi-Track Finder - #24

Open
dhshin04 wants to merge 13 commits into
devfrom
feat/plan-01-multi-track-loss-data
Open

Implement Single-Head Multi-Track Finder#24
dhshin04 wants to merge 13 commits into
devfrom
feat/plan-01-multi-track-loss-data

Conversation

@dhshin04

@dhshin04 dhshin04 commented May 23, 2026

Copy link
Copy Markdown
Collaborator

Summary

Implements single-run, single-head multi-track loss and data pipeline improvements for MultiTrackFinder.

  • models/losses.py: Add min_perm_multi_track_loss — pure-TensorFlow min-over-permutations matching. Evaluates all P! GT reorderings, minimizes masked sparse CE + focal presence per permutation, and adds an inter-pair diversity penalty outside the min. Supports max_pairs 1–5 (hard cap at 5). Old multi_track_loss kept for A/B comparison.
  • data/multi_track/gen_training_random.py: Canonical GT pair ordering (sort by μ⁺ element ID sum) and --pair_dist skewed option (exponentially decaying pair-count distribution).
  • models/MultiTrackFinder.py: Wire min_perm_multi_track_loss(max_pairs=...); add --focal_gamma and --lambda_diversity CLI args; fix --lambda_presence default to 1.0.
  • scripts/train_multi_v2.slurm: New training script with updated hyperparameters (lambda_presence=1.0, focal_gamma=2.0, lambda_diversity=0.05), outputs multi_track_finder_v2.keras.
  • scripts/preprocess_multitrack.slurm: Add --pair_dist skewed --at_least_one_pair to train and val data generation.
  • eval_multi_track.py: Apply canonical GT sort before residual computation so eval ordering matches training.
  • docs/plan/: Updated Plan 01, 03, and 00_EXECUTION_ORDER.md.

Before Running train_multi_v2.slurm

Reprocess training data first (canonical ordering + skewed pair distribution):

sbatch scripts/preprocess_multitrack.slurm

Recommended: 1-epoch multi-GPU smoke test on the cluster before the full 72h run:

# Temporarily add --epochs 1 to the python3 call in train_multi_v2.slurm
sbatch scripts/train_multi_v2.slurm

Verify finite loss on epoch 1, then restore --epochs 60 for the full run.

Test plan

  • Single-device smoke test: min_perm_multi_track_loss() returns finite value for (4, 5, 2, 62) batch
  • Pre-commit hooks (ruff lint + format) pass on changed files
  • Code review: shape/broadcast fix for gather+transpose; max_pairs parameterized with P<=5 guard
  • Multi-GPU smoke test on HPC cluster before full training run
  • After merge: rerun preprocessing, then sbatch scripts/train_multi_v2.slurm

@dhshin04
dhshin04 changed the base branch from main to dev May 24, 2026 01:32
@dhshin04 dhshin04 changed the title Plan 01: Multi-track loss & data pipeline improvements Implement Single-Head Multi-Track Finder Jun 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant