Skip to content

Allow LeRobot imports to publish into bucket-backed HFlow data roots #304

Description

@kstonekuan

Current behavior

hflow import lerobot parses --output-dir as a Path and describes it as a local destination at src/hflow/cli.py:297-302. The public importer also accepts output_dir: Path, keeps _lerobot_cache below it, and returns list[Path] at src/hflow/importers/lerobot.py:500-550.

The rest of HFlow treats a data root as local storage or s3://, gs://, and az:// through parse_storage_root() and the shared publish/fetch boundary at src/hflow/storage.py:630-655. A bucket-backed workspace currently needs an extra manual upload step after LeRobot import.

Definition of done

  • The public importer and CLI accept every data-root form supported by parse_storage_root().
  • Existing local-path behavior and CLI usage remain compatible.
  • For a bucket destination, Hugging Face downloads, video slices, and MCAP construction use local staging. Source cache files aren't uploaded into the durable data root.
  • Canonical episodes publish under landing/ through the existing StorageRoot boundary.
  • prepared-manifest.json publishes after every selected episode object succeeds.
  • The public result reports real remote object URIs for bucket outputs. It doesn't represent a remote object as a Path.
  • A failed upload leaves no partial remote object or manifest claiming that the import completed.
  • Missing bucket dependencies and provider authentication failures reach the CLI as concise errors with no traceback.
  • Add behavioral coverage for both local and bucket-backed destinations using the existing storage test boundary.
  • Update docs/how-to/import-lerobot-v3.md with one bucket example and explain where the local cache lives.

Batch retry and episode reuse are tracked separately in #303. The two designs should compose, but this issue doesn't add resumable imports itself.

Non-goals

  • Airflow execution
  • Parallel episode conversion
  • Bucket or filesystem watchers
  • A new storage backend or package
  • Changes to the canonical MCAP layout

Validation

uv run ruff check
uv run ruff format --check
uv run ty check
uv run pytest tests/test_lerobot_converter.py tests/test_storage.py -q
uv run pytest -q

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    advancedNeeds codebase familiarity; not a starter issueenhancementNew feature or requesthelp wantedExtra attention is needed

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions