Current behavior
LeRobot Dataset v3 episode metadata may be split across files below meta/episodes/. The importer downloads each tree entry to its basename at src/hflow/importers/lerobot.py:331-341.
That flattens paths such as:
meta/episodes/chunk-000/file-000.parquet
meta/episodes/chunk-001/file-000.parquet
Both become _lerobot_cache/meta/episodes/file-000.parquet. The second entry sees the first file and skips its download. The same local file is then appended twice and read twice.
There is a second first-shard assumption at src/hflow/importers/lerobot.py:379-438. The importer discovers video columns and reads video-window rows only from episode_metadata_files[0], so episodes from later metadata files get no indexed video window.
Verified on main at 1f5c7c8. The current synthetic corpus has one metadata file, so neither outcome is covered.
Pattern to copy
Extend test_index_discovery_multi_camera_metadata. It already builds v3 metadata, replaces the Hugging Face network boundary, and asserts parsed episode and camera outcomes.
The data-file cache in src/hflow/importers/lerobot.py:656-668 keeps chunk and file identities distinct.
What to build
Preserve enough of each meta/episodes tree path that files from different chunks cannot collide. Read episode rows and video windows from every downloaded metadata file.
Add a synthetic corpus with at least two metadata shards. Use the same basename in different chunk directories, put a different episode in each shard, and give each episode a distinct video window.
Definition of done
- Metadata files with the same basename in different chunk directories use different local paths.
- Every metadata shard is downloaded or reused from its own cache entry.
- Episode rows from all shards appear exactly once and remain sorted by
episode_index.
- Video chunk, file, and timestamp windows from every shard are attached to the matching episode.
- The regression fails against current
main and passes with the fix.
- The one-shard and multi-camera behavior stays unchanged.
- Bump
CONVERTER_VERSION if the fixed case can currently publish different canonical bytes.
- No new runtime dependency is added.
Validation
uv run ruff check --fix
uv run ruff format
uv run ty check
uv run pytest -q tests/test_lerobot_converter.py
uv run pytest -q
Run the full suite from the repo root. pytest tests alone misses packages/hflow-server/tests.
Current behavior
LeRobot Dataset v3 episode metadata may be split across files below
meta/episodes/. The importer downloads each tree entry to its basename atsrc/hflow/importers/lerobot.py:331-341.That flattens paths such as:
Both become
_lerobot_cache/meta/episodes/file-000.parquet. The second entry sees the first file and skips its download. The same local file is then appended twice and read twice.There is a second first-shard assumption at
src/hflow/importers/lerobot.py:379-438. The importer discovers video columns and reads video-window rows only fromepisode_metadata_files[0], so episodes from later metadata files get no indexed video window.Verified on
mainat1f5c7c8. The current synthetic corpus has one metadata file, so neither outcome is covered.Pattern to copy
Extend
test_index_discovery_multi_camera_metadata. It already builds v3 metadata, replaces the Hugging Face network boundary, and asserts parsed episode and camera outcomes.The data-file cache in
src/hflow/importers/lerobot.py:656-668keeps chunk and file identities distinct.What to build
Preserve enough of each
meta/episodestree path that files from different chunks cannot collide. Read episode rows and video windows from every downloaded metadata file.Add a synthetic corpus with at least two metadata shards. Use the same basename in different chunk directories, put a different episode in each shard, and give each episode a distinct video window.
Definition of done
episode_index.mainand passes with the fix.CONVERTER_VERSIONif the fixed case can currently publish different canonical bytes.Validation
Run the full suite from the repo root.
pytest testsalone missespackages/hflow-server/tests.