feat(env): manage env state centrally

[jx-qiu]

This pull request updates the documentation to reflect recent changes in how environment state, tool registration, and artifact management are handled in the codebase. The main focus is on the transition from manual tool registration and file management to an automated, EnvState-centric workflow. Documentation in both English and Chinese is revised for clarity and accuracy, and references to legacy interfaces and files are replaced with the new conventions.

Major documentation and interface updates:

General workflow and tool registration:

• Updated instructions for adding scripted primitives: tool registration is now automatic based on TOOLS_SPEC and @readonly. No need to manually call add_tools anymore! [1] [2] [3] [4]
• Clarified that tool return values should be small dicts, and large observations must be saved via EnvState.save; images and geometry are exposed through dedicated tools, not raw paths. [1] [2]

State management and artifact handling:

• Revised descriptions of per-step state dumps: now handled by EnvState.record_step and EnvState.save, with explicit artifact tracking. Artifacts are referenced by logical base names, not manual paths. [1] [2]
• Updated conventions to emphasize that all environment observations are owned and managed by EnvState, and that callers must not construct storage paths directly. [1] [2]

Tool interface and naming consistency:

• Standardized on view_env_state instead of view_driver_state throughout the documentation and interface references. [1] [2] [3] [4] [5] [6]

Quickstart and calibration documentation:

• Updated quickstart guides to explain the new artifact naming, storage, and retrieval conventions, including the use of view_env_state(step=-1) for inspecting the latest state. [1] [2]
• Improved calibration guide instructions to use view_env_state({"step": 0}) for initial state and clarified that direct file or BDDL access is not supported. [1] [2]

Code cleanup:

• Removed the unused video_path parameter from get_toolkit in robots/libero/__init__.py.

[Copilot]

Pull request overview

This PR centralizes environment step recording and artifact management around a new EnvState/StepRecord model, and updates the LIBERO runtime + documentation to use the new view_env_state workflow instead of legacy file-path/state-dump conventions.

Changes:

• Introduces EnvState (manifest + artifact IO) and @updatestate to automate post-tool state capture/publishing.
• Updates LIBERO tools/toolkit and dashboard to consume step records/artifacts (and standardizes on view_env_state naming).
• Refreshes English/Chinese docs and guides to match the new artifact naming + retrieval contract.

Reviewed changes

Copilot reviewed 30 out of 30 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
scripts/codex_proxy/litellm_callbacks.py	Updates doc/example namespace map to use view_env_state.
rpent/utils/sam3_client.py	Changes SAM3 client API to accept image bytes/arrays instead of paths.
rpent/tools/toolkit.py	Adds @updatestate marker and state-capture/publish flow for stateful tools.
rpent/tools/state.py	Adds new EnvState + StepRecord implementation (manifest + artifact storage).
rpent/tools/common.py	Adds shared backprojection/centroid helpers used by tools.
rpent/planner/api_loop.py	Updates planner helper text to reference view_env_state.
rpent/dashboard/state.py	Adds StepRecordEvent handling and loads frames/action videos via EnvState.
rpent/dashboard/server.py	Switches video endpoints to return in-memory bytes responses.
rpent/dashboard/events.py	Introduces StepRecordEvent for step publication to dashboard.
rpent/context/prompts/prompt.py	Updates prompt guidance from view_driver_state to view_env_state.
rpent/cli/main.py	Removes legacy video_path wiring into toolkit creation.
rpent/cli/dashboard.py	Removes legacy video_path wiring into toolkit creation (dashboard mode).
robots/libero/tools.py	Migrates LIBERO tool implementations to EnvState artifacts + view_env_state.
robots/libero/toolkit.py	Registers tools via TOOLS_SPEC + @updatestate; uses EnvState for recording.
robots/libero/prompts/user.py	Updates user prompt to instruct view_env_state and embedded images.
robots/libero/prompts/system.py	Rewrites system guidance around EnvState artifacts + view_env_state.
robots/libero/guides/strict_hybrid_guide.md	Replaces legacy file-path protocol with EnvState-centric runtime contract.
robots/libero/guides/pro_hybrid_guide.md	Updates PRO guide to the new view_env_state/EnvState workflow.
robots/libero/guides/env_calibration.md	Updates calibration doc to use structured tools + view_env_state access.
robots/libero/init.py	Removes video_path parameter from get_toolkit.
docs/source-zh/rst_source/usage/libero.rst	Chinese docs: rename view_driver_state → view_env_state + defaults.
docs/source-zh/rst_source/quickstart.rst	Chinese quickstart: clarifies states.json is EnvState manifest; avoid parsing.
docs/source-zh/rst_source/development/interfaces.rst	Chinese dev docs: references view_env_state.
docs/source-zh/rst_source/development/add_robot.rst	Chinese dev docs: documents EnvState.record_step/save conventions.
docs/source-zh/rst_source/development/add_primitive.rst	Chinese dev docs: documents @updatestate + automatic tool registration.
docs/source-en/rst_source/usage/libero.rst	English docs: rename view_driver_state → view_env_state + defaults.
docs/source-en/rst_source/quickstart.rst	English quickstart: clarifies states.json is internal EnvState manifest.
docs/source-en/rst_source/development/interfaces.rst	English dev docs: references view_env_state.
docs/source-en/rst_source/development/add_robot.rst	English dev docs: documents EnvState.record_step/save conventions.
docs/source-en/rst_source/development/add_primitive.rst	English dev docs: documents @updatestate + automatic tool registration.

Suppressed comments (1)

robots/libero/tools.py:793

• write_recipe_from_states always returns recipe_name even if EnvState.save(...) fails (it returns None on error). That can lead callers to believe a recipe was exported when it wasn't. Check the save result and raise (or otherwise report) on failure.

    state.save(
        recipe_name,
        [command for _, command in command_events],
        step=None,
    )

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Sonorous281]

Could we keep the previous retention policy for high-res artifacts here? Without cleanup, the two 1024×1024 world maps alone add 12 MB per step, which may become significant for long or batched rollouts.

[jx-qiu]

For libero, each step produces ~17 MiB of on-disk artifacts. I will see if there are opportunities for better lossless compression.

[wilburx813]

reset every file expect log seems unsafe, the user may accidentally pass a important directory

[wilburx813]

could we write manifest once per step instead of rewriting it after every artifact save?

[wilburx813]

btw, seems rpent-memory needs update too.

[jx-qiu]

btw, seems rpent-memory needs update too.

Do you mean the markdown files under resources/libero/memory?

[wilburx813]

btw, seems rpent-memory needs update too.

Do you mean the markdown files under resources/libero/memory?

Yes, some files under resources/libero/memory still reference something like view_driver_state