Real-corpus friction (Egocentric-10K/100K): no WebDataset ingest path; the video adapter's JPEG detour costs 2.3x landing size and a double lossy transcode for H.265 sources

[rakesh0x]

Context

I ran HFlow against the shape of builddotai/Egocentric-10K and builddotai/Egocentric-100K (the largest real egocentric factory corpora on Hugging Face, and the same source the examples/egocentric quickstart shard comes from). Both corpora are WebDataset-style tars of ~180 s H.265/MP4 clips (1080p30 for 10K, 456x256 for 100K) plus per-clip JSON sidecars and per-worker intrinsics.json.

The finding: there is no first-party ingestion path for this corpus shape, and the only available video-to-MCAP adapter detours through JPEG, which costs roughly 2.3x landing size and a double lossy transcode for H.265 sources.

What exists today

• src/hflow/importers/ contains only the LeRobot v3 converter. WebDataset-style tars have no importer; users must write their own converter (the documented escape hatch) or reach for hflow.testing.write_video_episode.
• write_video_episode is explicitly a demo/test helper, and it decodes the source video and re-encodes every frame as JPEG sensor_msgs/CompressedImage (_build_video_camera_messages in src/hflow/testing.py). The canonical transform then re-encodes that JPEG into in-band H.264.

So for an H.265 corpus the media is transcoded HEVC -> JPEG -> H.264 (two lossy steps, one avoidable) even though H.264 is the canonical target and ffmpeg decodes HEVC directly.

Measurements

Profile-matched HEVC media (ffmpeg testsrc2, same resolution/fps/codec as the corpora; the real footage is gated so I could not download it for this test). Apple Silicon macOS, ffmpeg 8.1.1 from Homebrew, hflow at 78a6b1f.

30 s @ 1920x1080 @ 30 fps (Egocentric-10K profile):

Step	Result
Source HEVC bytes (same span)	~7 MB
write_video_episode landing MCAP (JPEG)	55.3 MB (~8x source bitrate)
Canonical episode H.264 media bytes	24.5 MB (landing is 2.3x this)
Adapter wall time	1.8 s
Cold full pipeline (app.test)	12.0 s (~2.5x slower than realtime)
of which camera_frame_stats decode	8.1 s

60 s @ 456x256 @ 30 fps (Egocentric-100K profile): cold full pipeline 2.9 s (~48 ms per source-second) -- the small profile is fine; the cost is concentrated at 1080p.

At 2.5x-slower-than-realtime, a single process QC pass over Egocentric-10K's 10,000 hours is on the order of a machine-year; the per-episode unit cost is what matters, and most of it is avoidable decode/transcode overhead plus JPEG round-tripping.

Why it matters

1. Disk: wrapping a real corpus through the JPEG detour multiplies landing storage by ~2.3x versus what the canonical stage needs, before any pipeline run.
2. Quality: every real frame passes through two lossy re-encodes; camera_frame_stats and any downstream consumer measure artifacts of the JPEG step, not just of the source.
3. Awkwardness: the JSON sidecars (factory/worker/duration/fps/codec) map naturally onto episode/v1 metadata records and intrinsics.json onto calibration attachments, but there is no example or helper wiring a WebDataset tar into that, so every team rediscovers the adapter path on their own.

Suggestions (non-prescriptive)

• A first-party WebDataset-to-canonical importer (or a documented example next to examples/egocentric) that transcodes HEVC -> in-band H.264 directly, skipping the JPEG landing detour.
• Or, if the JPEG path is intentional for test isolation, a note in the docs saying so and pointing real-corpus users at the converter how-to, so the demo helper does not become the de facto ingestion path.

Happy to split any of this into smaller scoped issues if maintainers prefer.

[kstonekuan]

Thanks for the report.

Verified the two claims that carry it, on 8840f0d:

• src/hflow/importers/ has __init__.py and lerobot.py, nothing else.
• write_video_episode renders frames with ffmpeg's mjpeg encoder and wraps them as sensor_msgs/msg/CompressedImage with format jpeg (testing.py:298, :328, :481, :511, and _encode_compressed_image at :234). The canonical transform then encodes that into in-band H.264. So HEVC to JPEG to H.264 is exactly what happens.

examples/egocentric/prepare.py:452 calls hflow.testing.write_video_episode itself. Our own worked example for this corpus family goes through the JPEG detour, so it isn't a path users wander onto, it's the one we show them.

The JPEG choice inside testing.py is deliberate and I wouldn't change it there. That module's first line calls itself a fixture "for tests and demos", and its contract is to be HFlow's input: written with the stock MCAP writer, depending on nothing from the canonical writer (there's an architecture test pinning that now, tests/test_module_boundaries.py). Emitting ordinary CompressedImage JPEG is what makes it a realistic recorder output. A fixture that emitted in-band H.264 would be testing less.

The real problem is what follows from that: a helper whose job is to be plausible test input is the documented route for a real corpus. Your measurements are the cost of that mismatch, and I read them as profile-matched synthetic, which is how you presented them.

One question. Your 1080p numbers put 8.1s of a 12.0s pipeline in camera_frame_stats decode. How much of that survives if the landing episode is already H.264, so the check decodes one stream and not a JPEG-derived re-encode? If the decode cost is mostly independent of the detour, that's a separate issue.

[rakesh0x]

I reasoned through this rather than measuring it, so take it as analysis for now:

I think almost all of the 8.1s survives. camera_frame_stats never sees the JPEGs. By the time the check runs, the transform has already re-encoded everything to in-band H.264, so in both paths the check is decoding the same thing: 900 H.264 frames at 1080p30. The detour's extra cost sits upstream of the check, in the adapter's mjpeg encode plus the transform's jpeg to h264 re-encode, which is roughly the other 3.9s of the 12.0s run. A check that measures per frame luma to find blackouts and freezes cannot skip the decode without changing what it guarantees, so the 8s looks like the price of the measurement itself, not something the ingest path caused.

So I agree it reads as a separate issue, something like camera_frame_stats decode throughput at 1080p. That way the transcode cost stays with this issue and the check cost gets discussed on its own terms (decode parallelism, hwaccel, sampling, whatever fits the guarantee).

If a measured number is worth having in the thread I can run the control: the same 30s 1080p30 clip wrapped as native in-band H.264 CompressedVideo, which the transform passes through without re-encoding. Should be ten minutes of work.

@kstonekuan

[kstonekuan]

Makes sense. camera_frame_stats "requires a canonical episode" (checks.py:587), so it only runs after the transform. Both paths decode the same 900 frames of in-band H.264 and it never sees a JPEG. Detour cost sits upstream of it, as you have it.

Yes to the control run if it's really ten minutes. Cheap to settle, and analysis everyone agrees with is still the kind of thing that turns out wrong once. Publish the split: same clip wrapped as native in-band H.264 CompressedVideo so the transform passes it through, against the JPEG path, camera_frame_stats time called out separately from total pipeline time in both. If check time is flat across the two, that's the whole argument in one table.

Then open the decode-throughput issue yourself with the control in it. Frame it around what the check guarantees: per-frame luma isn't optional for finding blackouts and freezes, so the question is whether the same guarantee can be met with less decoding.

Leave this issue for the ingest path.

[kstonekuan]

Decision on the ingestion half: no first-party WebDataset importer.

WebDataset is a packaging convention, not a format with a spec. Every corpus inside a tar has its own sidecars, its own layout, its own metadata keys. The LeRobot v3 importer generalises because v3 is a specified layout; a webdataset.py next to it would really be an Egocentric-10K importer wearing a generic name, and we'd owe one per corpus shape after that. Writing your own converter stays the answer for a new corpus, and that's the documented escape hatch.

What does change is the two things that made you file this:

1. examples/egocentric/prepare.py stops calling hflow.testing.write_video_episode and transcodes HEVC to in-band H.264 directly. That removes the JPEG round-trip, the second lossy step, and the ~2.3x landing size, and it makes the example a converter worth copying instead of a fixture being borrowed.
2. The docs say plainly that write_video_episode is a test fixture and point real ingestion at docs/how-to/write-a-converter.md.

I'm filing (1) now and will link it here. Your measurements go in it as the before numbers.

Still interested in the control run and the decode-throughput issue from the thread above, both independent of this.

[kstonekuan]

Filed as #291.