diff --git a/docs/source-en/rst_source/development/add_primitive.rst b/docs/source-en/rst_source/development/add_primitive.rst index dcefdd5b..bb52a8b2 100644 --- a/docs/source-en/rst_source/development/add_primitive.rst +++ b/docs/source-en/rst_source/development/add_primitive.rst @@ -35,7 +35,7 @@ call. They differ only in how the method is implemented. Add a scripted primitive ------------------------ -Adding a scripted primitive usually involves three steps: +Adding a scripted primitive usually involves two steps: 1. **Add a method to the primitives.** Add the method to the current environment's primitives class, such as @@ -43,6 +43,9 @@ Adding a scripted primitive usually involves three steps: the tool-call arguments, performs the work, usually through one or more ``self._env.step(...)`` calls, and returns a small log ``dict``. + Primitive methods capture and re-render state (``get_env_state``) + automatically after they run: + .. code-block:: python def open_drawer(self, dx: float = 0.15) -> dict: @@ -51,8 +54,12 @@ Adding a scripted primitive usually involves three steps: self._env.step(build_open_drawer_chunk(dx)) return {"ok": True, "dx": dx} + You can mark read-only tools (``view_env_state``, ``back_project``, ``segment``, + ...) with :func:`~rpent.tools.toolkit.readonly` so the toolkit skips state + capture for them, improving performance. + 2. **Add the tool schema.** Add an entry to ``TOOLS_SPEC`` in - ``toolkit.py``: + ``robots//tools.py``: .. code-block:: python @@ -67,13 +74,9 @@ Adding a scripted primitive usually involves three steps: }, } -3. **Register the tool in the toolkit.** Route it through the toolkit's - ``_step`` helper so that state is re-rendered after execution: - - .. code-block:: python - - self.add_tool("open_drawer", OPEN_DRAWER_SPEC, - lambda **kw: self._step("open_drawer", **kw)) +Once both exist, the toolkit registers the tool automatically: it iterates +``TOOLS_SPEC`` and binds each spec to the matching primitive-driver method +(e.g. ``getattr(self._primitives, name)``). After these steps, the ``api``, ``claude_code``, and ``codex`` planners can all call the primitive without any other code changes. @@ -167,8 +170,10 @@ Design principles for a new primitive the state dump reflecting the post-action world. Don't let the primitive return before the render finishes. - **Return small dicts.** Tool return values are fed back to the LLM - as text. Store larger content, such as images, depth data, and - ``states.json``, in the state dump instead. + as text. Save larger observations through ``EnvState.save``; ``EnvState`` + automatically records each logical base name in its owned + ``StepRecord.artifacts`` set. Expose images through ``view_env_state`` and + geometry through environment tools rather than returning raw paths. - **Guardrails belong in env_server**, not in the toolkit. The LLM can and will call any tool with any arguments; workspace bounds and safety clamps must be enforced on the server side. diff --git a/docs/source-en/rst_source/development/add_robot.rst b/docs/source-en/rst_source/development/add_robot.rst index 2111a9f7..d6e56ecd 100644 --- a/docs/source-en/rst_source/development/add_robot.rst +++ b/docs/source-en/rst_source/development/add_robot.rst @@ -220,23 +220,29 @@ state needed for the current run. It exposes one method per primitive tool **Tool definitions and handlers** — a module-level ``TOOLS_SPEC`` list of Anthropic-style tool definitions (``name``, ``description``, ``input_schema``), plus any module-level functions referenced by the toolkit (e.g. -``view_driver_state``, ``back_project``, ``finish``). - -**Per-step state dump** — ``dump_state(primitives, output_dir, step_idx, log)`` -serializes whatever state the agent will read back via the ``view_*`` tools -(images, depths, JSON state, camera meta) into ``output_dir``. +``view_env_state``, ``back_project``, ``finish``). + +**Per-step state dump** — ``dump_state(driver, env_state, log)`` opens +``env_state.record_step(...)`` and receives the allocated step index; the +``StepRecord`` is appended and committed immediately. Save large observations +through ``env_state.save(...)`` — inside a ``record_step`` block the ``step`` +argument may be omitted (it defaults to the new step), pass an explicit +``step=`` to target a different step, and ``step=None`` for run-level +artifacts. ``EnvState`` adds every successfully saved base name to the step's +flat ``artifacts`` set automatically. Readers use the canonical artifact +filenames rather than maintaining a parallel observation index. **Toolkit class** — subclass ``rpent.tools.toolkit.Toolkit``: - build the primitives in ``__init__`` through a custom initialization - helper (named ``init_primitives_clean`` in LIBERO; it wipes stale - ``images/`` etc., constructs the primitives, and dumps step 0), + helper (named ``init_primitives_clean`` in LIBERO; it calls + ``EnvState.reset()``, constructs the primitives, and dumps step 0), - register each tool with ``self.add_tool(name, spec, handler)`` — stateless - readers (``view_driver_state``, ``finish``, …) bind directly to module-level + readers (``view_env_state``, ``finish``, …) bind directly to module-level functions; primitive tools route through ``_step(name, **kwargs)`` which calls ``getattr(self._primitives, name)(**kwargs)`` and re-renders state, -- override ``close()`` to write any remaining agent-side artifacts (e.g. the - LIBERO toolkit saves the agentview MP4 there). +- override ``close()`` to save remaining agent-side artifacts through + ``EnvState`` (for example ``state.save("episode.mp4", frames, step=None)``). ``primitives_kwargs`` (forwarded from ``__init__.py:get_toolkit``) is the dict the toolkit passes verbatim to your primitives' ``__init__`` — typically @@ -246,14 +252,15 @@ Conventions worth keeping ------------------------- - ``output_dir`` is the working directory that the runner creates for each - run. Images, depths, ``states.json``, transcripts, ``episode.mp4``, and other - artifacts go there. + run. Environment observations are owned by ``EnvState``; callers use logical + base names and never construct storage paths. Transcripts and other + run-management outputs share the same run directory. - Tool definitions use the Anthropic format (``name`` / ``description`` / ``input_schema``). Every tool registered with ``self.add_tool(...)`` is exposed to all planners. - Server-side return values must be picklable and torch-free. - Each primitive tool dumps a fresh state snapshot after running so the next - ``view_driver_state`` call reflects the post-action world. + ``view_env_state`` call reflects the post-action world. - Treat ``dump_state`` as the source of truth for what the agent sees — any new modality (e.g. tactile, force) goes through it. diff --git a/docs/source-en/rst_source/development/interfaces.rst b/docs/source-en/rst_source/development/interfaces.rst index bf6a6a59..02d865a0 100644 --- a/docs/source-en/rst_source/development/interfaces.rst +++ b/docs/source-en/rst_source/development/interfaces.rst @@ -90,7 +90,7 @@ Subclass ``Toolkit`` in ``robots//toolkit.py`` and register env tools with ends; optional ``_image_bytes`` (etc.) to return camera images. The base class already registers common file tools; call ``super().__init__()`` then -``add_tool`` for env tools. Per-step state and ``view_driver_state`` are in +``add_tool`` for env tools. Per-step state and ``view_env_state`` are in :doc:`add_primitive`. Inter-process communication diff --git a/docs/source-en/rst_source/quickstart.rst b/docs/source-en/rst_source/quickstart.rst index b9ba30b5..05d60d5a 100644 --- a/docs/source-en/rst_source/quickstart.rst +++ b/docs/source-en/rst_source/quickstart.rst @@ -103,14 +103,11 @@ A successful run: by the elapsed time, token usage, and path to the run record. 3. With the Dashboard enabled, also streams agent output, camera views, the action timeline, and clip replays to the Dashboard. -4. By default, artifacts are saved under - ``logs/__t_s/``. They include - ``transcript_*.json`` (run record), ``states.json`` (one record per - environment step), ``recipe_*.jsonl`` (action sequence), and - ``episode.mp4`` (episode video). - -After the run, inspect the final record in ``states.json``: -``libero_terminated`` set to ``true`` means LIBERO judged the task complete. -You can also open ``episode.mp4`` to review the run. +4. By default, artifacts are saved under ``logs/__t_s/``. They include ``transcript_*.json`` (run record), ``states.json`` (the ``EnvState`` manifest), ``recipe_*.jsonl`` (action sequence), and ``episode.mp4`` (episode video). Each step artifact has a directory named after its logical artifact name; zero-padded step files live inside it, for example ``agentview_depth.npy/00.npy`` and ``agentview_depth.npy/01.npy``. Run-level artifacts remain at the output root. + +Inspect the final state through the Dashboard or +``view_env_state(step=-1)``. Its top-level ``terminated`` value is the +benchmark outcome. ``states.json`` is internal ``EnvState`` storage and should +not be parsed by callers. You can also open ``episode.mp4`` to review the run. If something goes wrong, inspect the four log files described at the bottom of :doc:`installation`. diff --git a/docs/source-en/rst_source/usage/libero.rst b/docs/source-en/rst_source/usage/libero.rst index 4fd25442..7e90c120 100644 --- a/docs/source-en/rst_source/usage/libero.rst +++ b/docs/source-en/rst_source/usage/libero.rst @@ -116,7 +116,7 @@ What runs where transports (HTTP or socket). It returns only the top compressed PNG mask. - **toolkit** (``robots/libero/toolkit.py``) — defines the tools the LLM can call: ``pi0_pick`` (fed to Pi0.5), ``move_to``, - ``rotate_wrist``, ``back_project``, ``view_driver_state``, + ``rotate_wrist``, ``back_project``, ``view_env_state``, ``finish``, … Tools the planner can call @@ -147,8 +147,10 @@ Physical action tools advance the environment and record new state and images. coordinates. - ``segment(prompt=... / point=..., ...)`` — use SAM3 to segment an existing image with a text or point prompt. -- ``view_driver_state(step=None)`` — read an existing state and image record. -- ``view_camera_meta(camera=..., step=None)`` — read existing camera metadata. +- ``view_env_state(step=-1)`` — read a recorded state and its embedded + observation images. Step ``0`` is initial; ``-1`` is latest. +- ``view_camera_meta(camera=..., step=-1)`` — read camera metadata for a + recorded step. Step ``-1`` is latest. - ``finish(status, summary)`` — end the current run. These tools do not advance the environment. diff --git a/docs/source-zh/rst_source/development/add_primitive.rst b/docs/source-zh/rst_source/development/add_primitive.rst index 909a7e82..61288934 100644 --- a/docs/source-zh/rst_source/development/add_primitive.rst +++ b/docs/source-zh/rst_source/development/add_primitive.rst @@ -33,13 +33,16 @@ primitives 方法,以及调用完成后的状态快照。区别仅在于方法 添加一个脚本化原语 ------------------ -添加脚本化原语通常需要以下三个步骤: +添加脚本化原语通常需要以下两个步骤: 1. **在 primitives 中添加方法。** 在当前环境的 primitives 类(如 ``LiberoPrimitives``、``MyRobotPrimitives``)中添加 一个方法。该方法接收工具调用的参数,执行一次或多次 ``self._env.step(...)``,并返回一个简短的日志字典。 + primitive 方法执行后默认会自动捕获并重新渲染状态 + (``get_env_state``): + .. code-block:: python def open_drawer(self, dx: float = 0.15) -> dict: @@ -48,7 +51,11 @@ primitives 方法,以及调用完成后的状态快照。区别仅在于方法 self._env.step(build_open_drawer_chunk(dx)) return {"ok": True, "dx": dx} -2. **添加工具定义。** 在 ``toolkit.py`` 的 ``TOOLS_SPEC`` 中新增一项: + 只读工具(``view_env_state``、``back_project``、``segment`` 等) + 可以使用 :func:`~rpent.tools.toolkit.readonly` 标记,toolkit 会跳过 + 它们的状态捕获,提升性能。 + +2. **添加工具定义。** 在 ``robots//tools.py`` 的 ``TOOLS_SPEC`` 中新增一项: .. code-block:: python @@ -63,13 +70,8 @@ primitives 方法,以及调用完成后的状态快照。区别仅在于方法 }, } -3. **在 toolkit 中注册工具。** 通过 toolkit 的 ``_step`` 辅助函数运行 - 该工具,使其在执行结束后自动重新渲染状态: - - .. code-block:: python - - self.add_tool("open_drawer", OPEN_DRAWER_SPEC, - lambda **kw: self._step("open_drawer", **kw)) +两者就位后,toolkit 会自动注册该工具:它遍历 ``TOOLS_SPEC``,把每个定义 +绑定到对应的 primitive 方法(如 ``getattr(self._primitives, name)``)。 完成以上步骤后,``api``、``claude_code`` 和 ``codex`` 三种 planner 都可以调用该工具,无需修改其他代码。 @@ -152,7 +154,9 @@ primitives 方法,以及调用完成后的状态快照。区别仅在于方法 - **每个工具执行结束后都要保存新的状态快照。** 下一轮需要读取动作执行后的 环境状态,因此原语不能在渲染完成前返回。 - **工具只返回简短的字典。** 返回值会以文本形式提供给 LLM;图像、深度数据和 - ``states.json`` 等较大的内容则通过状态快照提供。 + 其他大型观测应通过 ``EnvState.save`` 保存;``EnvState`` 会把每个逻辑基础 + 文件名自动加入其持有的 ``StepRecord.artifacts`` 集合。图像通过 + ``view_env_state`` 提供,几何数据通过环境工具访问,不返回原始路径。 - **安全限制由 ``env_server`` 强制执行。** LLM 可能使用任意参数调用工具, 因此工作空间边界和安全限制不能只依赖 toolkit。 diff --git a/docs/source-zh/rst_source/development/add_robot.rst b/docs/source-zh/rst_source/development/add_robot.rst index 995a13e7..14eddb67 100644 --- a/docs/source-zh/rst_source/development/add_robot.rst +++ b/docs/source-zh/rst_source/development/add_robot.rst @@ -207,23 +207,27 @@ toolkit 模块通常包含四部分: **工具定义和处理函数** 包括模块级的 ``TOOLS_SPEC`` 列表(列表元素采用 Anthropic API 的工具定义格式,包含 ``name``、``description`` 和 ``input_schema``),以及 toolkit 引用的模块级函数,例如 -``view_driver_state``、``back_project`` 和 ``finish``。 +``view_env_state``、``back_project`` 和 ``finish``。 -**每步状态 dump** —— ``dump_state(primitives, output_dir, step_idx, log)`` 把 agent -之后会通过 ``view_*`` 工具读回的所有状态 (图像、深度、JSON 状态、camera meta) -序列化到 ``output_dir``。 +**每步状态 dump** —— ``dump_state(driver, env_state, log)`` 通过 +``env_state.record_step(...)`` 创建由 ``EnvState`` 持有的步骤,并取得分配的 +step index;该 ``StepRecord`` 会被立即追加并提交。大型观测通过 +``env_state.save(...)`` 保存——在 ``record_step`` 块内可省略 ``step`` 参数 +(默认指向刚创建的步骤),传显式 ``step=`` 可指定其它步骤,``step=None`` +用于运行级工件。每次保存成功后,``EnvState`` 会自动把基础文件名加入该 +``StepRecord`` 的扁平 ``artifacts`` 集合;读取方直接使用规范化的工件文件名。 **Toolkit 类** 继承 ``rpent.tools.toolkit.Toolkit``: - 在 ``__init__`` 中通过自定义的初始化辅助方法构建 primitives(LIBERO - 中的方法名为 ``init_primitives_clean``;它会清理过期的 ``images/`` 等目录、 - 构造原语并 dump 第 0 步), + 中的方法名为 ``init_primitives_clean``;它会调用 ``EnvState.reset()``、构造 + 原语并 dump 第 0 步), - 用 ``self.add_tool(name, spec, handler)`` 注册每个工具。无状态的读取工具 - (如 ``view_driver_state``、``finish``)直接绑定模块级函数;原语工具通过 + (如 ``view_env_state``、``finish``)直接绑定模块级函数;原语工具通过 ``_step(name, **kwargs)`` 调用。``_step`` 使用 - ``getattr(self._primitives, name)(**kwargs)`` 调用 primitives 方法并重新渲染状态; -- 重写 ``close()``,将 agent 侧生成的文件写入磁盘(例如 LIBERO toolkit - 在这里保存 agentview MP4)。 + ``getattr(self._primitives, name)(**kwargs)`` 调用 driver 方法并重新渲染状态; +- 重写 ``close()``,通过 ``EnvState`` 保存 agent 侧剩余工件(例如 + ``state.save("episode.mp4", frames, step=None)``)。 ``primitives_kwargs`` 由 ``__init__.py:get_toolkit`` 转发给 toolkit,再原样传入 primitives 的 ``__init__``。其中通常包含 @@ -232,14 +236,15 @@ primitives 的 ``__init__``。其中通常包含 建议遵循的约定 -------------- -- ``output_dir`` 是 runner 为单次运行创建的临时目录。图像、深度数据、 - ``states.json``、transcript 和 ``episode.mp4`` 等工件都写入该目录。 +- ``output_dir`` 是 runner 为单次运行创建的工作目录。环境观测由 + ``EnvState`` 管理;调用方只使用逻辑基础文件名,不自行拼接存储路径。 + transcript 等运行管理输出与环境工件共享该目录。 - 工具定义使用 Anthropic API 格式(``name`` / ``description`` / ``input_schema``)。 每个用 ``self.add_tool(...)`` 注册的工具都会暴露给所有 planner。 - 环境侧的返回值必须可 pickle,且不包含 torch 对象。 - 每个原语工具执行后要 dump 一次新的状态快照, 这样下一次 - ``view_driver_state`` 看到的是动作后的世界。 + ``view_env_state`` 看到的是动作后的世界。 - ``dump_state`` 是 Agent 获取环境状态的唯一数据来源;任何新的模态 (例如触觉、力)都通过它提供。 diff --git a/docs/source-zh/rst_source/development/interfaces.rst b/docs/source-zh/rst_source/development/interfaces.rst index d28c3d34..822b02fa 100644 --- a/docs/source-zh/rst_source/development/interfaces.rst +++ b/docs/source-zh/rst_source/development/interfaces.rst @@ -87,7 +87,7 @@ Planner 需要回传相机图时可设 ``_image_bytes`` 等字段。 基类已注册公共文件工具;子类 ``super().__init__()`` 后追加本环境工具即可。逐步状态与 -``view_driver_state`` 见 :doc:`add_primitive`。 +``view_env_state`` 见 :doc:`add_primitive`。 进程间通信 ---------- diff --git a/docs/source-zh/rst_source/quickstart.rst b/docs/source-zh/rst_source/quickstart.rst index b9cf71c7..4d4f14aa 100644 --- a/docs/source-zh/rst_source/quickstart.rst +++ b/docs/source-zh/rst_source/quickstart.rst @@ -95,7 +95,9 @@ LIBERO-PRO 仿真资源。下面以 LIBERO-PRO 和 ``claude_code`` planner 1. 终端会先显示 ``env_server``、``vla_server`` 和 ``sam3_server`` 的启动信息。 2. 智能体的逐轮输出和工具调用会显示在终端中;运行结束时还会显示耗时、token 用量和运行记录的路径。 3. 启用 Dashboard 后,智能体的输出、相机视图、动作时间线和片段回放也会实时显示在 Dashboard 中。 -4. 默认输出目录为 ``logs/__t_s/``,其中包含 ``transcript_*.json``\ (运行记录)、``states.json``\ (每个环境步的记录)、``recipe_*.jsonl``\ (动作序列)和 ``episode.mp4``\ (回合录像)。 +4. 默认输出目录为 ``logs/__t_s/``,其中包含 ``transcript_*.json``\ (运行记录)、``states.json``\ (``EnvState`` 清单)、``recipe_*.jsonl``\ (动作序列)和 ``episode.mp4``\ (回合录像)。每种逐步工件使用一个与逻辑工件同名的目录,目录内按步骤保存零填充文件,例如 ``agentview_depth.npy/00.npy`` 和 ``agentview_depth.npy/01.npy``;运行级工件仍保存在输出目录根部。 -运行结束后,查看 ``states.json`` 的最后一条记录:``libero_terminated`` 为 ``true`` 表示 LIBERO 已判定任务完成;也可以打开 ``episode.mp4`` 复核运行过程。 +通过 Dashboard 或 ``view_env_state(step=-1)`` 查看最终状态;其顶层 +``terminated`` 即为基准任务结果。``states.json`` 是 ``EnvState`` 的内部 +存储,调用方不应直接解析。也可以打开 ``episode.mp4`` 复核运行过程。 出问题时,参考 :doc:`installation` 页底部提到的四份日志文件。 diff --git a/docs/source-zh/rst_source/usage/libero.rst b/docs/source-zh/rst_source/usage/libero.rst index 2675458b..940de6da 100644 --- a/docs/source-zh/rst_source/usage/libero.rst +++ b/docs/source-zh/rst_source/usage/libero.rst @@ -112,7 +112,7 @@ LIBERO-PRO 核心套件一览 排名第一的压缩 PNG mask。 - **toolkit(工具集)** (``robots/libero/toolkit.py``)—— 定义 LLM 能调用的工具:``pi0_pick``(交给 Pi0.5)、``move_to``、``rotate_wrist``、 - ``back_project``、``view_driver_state``、``finish``… + ``back_project``、``view_env_state``、``finish``… Planner 能调用的工具 -------------------- @@ -141,8 +141,10 @@ LIBERO 工具分为物理动作工具和只读工具。 - ``back_project(row, col, ...)`` —— 将图像像素反投影到世界坐标。 - ``segment(prompt=... / point=..., ...)`` —— 通过 SAM3 对已有图像进行文本或 点提示分割。 -- ``view_driver_state(step=None)`` —— 读取已有的状态和图像记录。 -- ``view_camera_meta(camera=..., step=None)`` —— 读取已有的相机元数据。 +- ``view_env_state(step=-1)`` —— 读取已记录的状态和内嵌观测图像;第 0 步为 + 初始状态,``-1`` 表示最新状态。 +- ``view_camera_meta(camera=..., step=-1)`` —— 读取指定步骤的相机元数据; + ``-1`` 表示最新状态。 - ``finish(status, summary)`` —— 结束当前运行。 这些工具不会推进环境。 diff --git a/pyproject.toml b/pyproject.toml index 0200869a..3b47e229 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -58,6 +58,7 @@ openpi = [ ] libero = [ "rlinf-libero", + "h5py", ] libero-pro = [ "rpent[libero]", diff --git a/robots/libero/__init__.py b/robots/libero/__init__.py index c5f31923..48b1bcde 100644 --- a/robots/libero/__init__.py +++ b/robots/libero/__init__.py @@ -45,7 +45,6 @@ def get_toolkit( *, primitives_kwargs: dict[str, Any], dashboard_events: DashboardEventSink, - video_path: str | None = None, ): """Return the LIBERO toolkit (common tools + LIBERO primitives).""" from robots.libero.toolkit import LiberoToolkit @@ -53,7 +52,6 @@ def get_toolkit( return LiberoToolkit( primitives_kwargs=primitives_kwargs, dashboard_events=dashboard_events, - video_path=video_path, ) diff --git a/robots/libero/env_client.py b/robots/libero/env_client.py index 5c33776a..632c19ac 100644 --- a/robots/libero/env_client.py +++ b/robots/libero/env_client.py @@ -13,7 +13,6 @@ from rpent.utils.rpc import RpcClient - _TIMEOUT_S = { "default": 30.0, "env.reset": 120.0, @@ -35,8 +34,8 @@ def __init__( ): self._client = client self.return_all_frames = return_all_frames - self.episode_terminated = False - self.episode_truncated = False + self.terminated = False + self.truncated = False server_meta = self._client.call( "env.get_env_meta", timeout_s=_TIMEOUT_S["default"] ) @@ -49,17 +48,17 @@ def __init__( self.reset() def check_done(self, term, trunc) -> None: - self.episode_terminated |= bool(np.asarray(term).any()) - self.episode_truncated |= bool(np.asarray(trunc).any()) + self.terminated |= bool(np.asarray(term).any()) + self.truncated |= bool(np.asarray(trunc).any()) def reset(self) -> tuple[dict, Any]: ret = self._client.call("env.reset", timeout_s=_TIMEOUT_S["env.reset"]) - self.episode_terminated = False - self.episode_truncated = False + self.terminated = False + self.truncated = False return ret def step(self, action) -> tuple[dict, Any, np.ndarray, Any, Any]: - assert not (self.episode_terminated or self.episode_truncated), ( + assert not (self.terminated or self.truncated), ( "env.step called after the episode signaled term/trunc" ) ret = self._client.call( @@ -78,7 +77,7 @@ def chunk_step(self, actions, *, return_all_frames: bool | None = None) -> tuple Terminated / truncated have shape ``[chunk_size]`` after the server strips the env dim. """ - assert not (self.episode_terminated or self.episode_truncated), ( + assert not (self.terminated or self.truncated), ( "env.chunk_step called after the episode signaled term/trunc" ) if return_all_frames is None: diff --git a/robots/libero/guides/env_calibration.md b/robots/libero/guides/env_calibration.md index 1dbab00d..8b3a6240 100644 --- a/robots/libero/guides/env_calibration.md +++ b/robots/libero/guides/env_calibration.md @@ -3,11 +3,11 @@ ## Current LIBERO MCP Runtime Contract Use this file as a calibration reference only. For current MCP-based runs, use -structured MCP tools, do not issue file-based protocol commands, and do not manually manage -`env_server.py`. Do not read BDDL files or hidden task definition files to infer -coordinates. Do not expect object world coordinates in `states.json`; localize -objects through images_cam + depth/back_project, segment, and wrist/high-res -artifacts when available. +structured MCP tools, do not issue file-based protocol commands, and do not +manually manage `env_server.py`. Do not read BDDL files or hidden task +definition files to infer coordinates. Start with +`view_env_state({"step": 0})`; localize objects from `agentview_high.png` or +`wrist_high.png` through `back_project` or `segment`. Measured 2026-05-20 on `libero_10_with_mug` t0 (LIVING_ROOM frame) and t8 (KITCHEN frame). All probes use `move_to` with `gripper=-1` and tight @@ -17,9 +17,9 @@ Measured 2026-05-20 on `libero_10_with_mug` t0 (LIVING_ROOM frame) and t8 Each task scene uses one of the table fixtures below, which sets the entire world-frame z origin. The OSC workspace and all pick/place altitudes shift -accordingly. **Check `states.json[0].state.robot0_eef_pos[2]` in the initial -state and branch on it.** Do not read BDDL files for this; use runtime state and -visual evidence. +accordingly. **Check `state.robot0_eef_pos[2]` in the result of +`view_env_state({"step": 0})` and branch on it.** Do not read BDDL files for +this; use runtime state and visual evidence. | Fixture | eef home z | Table top z | Used by tasks | |---|---|---|---| @@ -119,7 +119,7 @@ limit 1.15). My libero_10 t0 used z=0.95 for travel — safe and consistent. ## Practical rules going forward -1. **Always read `states.json[0].state.robot0_eef_pos[2]` before computing any z target.** +1. **Always inspect the initial returned `state.robot0_eef_pos[2]` before computing any z target.** ≈ 0.68 → LIVING_ROOM; ≈ 1.17 → KITCHEN; ≈ 0.26 → OBJECT. Use the matching frame table above. 2. **Never command an eef z below the per-frame floor.** Going to z=0.42 @@ -130,7 +130,7 @@ limit 1.15). My libero_10 t0 used z=0.95 for travel — safe and consistent. 4. **`set_gripper` after a stalled `move_to` is unsafe.** The previous t0 attempt closed the gripper above the bottle (eef stalled high) then opened it; this returned ok but the env was effectively desynced. - Treat any `move_to` with `final_dist_m > 0.02` as a failure and + Treat any `move_to` with `log.result.final_dist_m > 0.02` as a failure and recover (back to safe altitude, re-plan) before proceeding. 5. **Drop height matters for basket tasks.** Release at eef z=0.58 in LIVING_ROOM frame caused basket displacement Δ≈4–5 cm; z=0.53 keeps @@ -143,13 +143,12 @@ limit 1.15). My libero_10 t0 used z=0.95 for travel — safe and consistent. `move_to` + `set_gripper` (last resort; unreliable for objects <6 cm — see `resources/libero/memory/feedback_scripted_pick_limits.md`). -## Calibration log files +## Calibration records -Raw probe logs are preserved in: -- `{output_dir}/states.json` (one step entry per command — the per-command - audit; each entry has `command`, `result`, `state`, `elapsed_s`). -- Only kept for the most recent agent session; reproduce by re-running - the calibration with the snippet in the next section. +Each calibration motion returns its state and `log` immediately. To revisit a +recorded step, call `view_env_state({"step": N})`; use `-1` for the latest. +The internal `states.json` file is a versioned `EnvState` manifest, not an +agent-facing list for manual indexing. ## Reproducer @@ -162,5 +161,5 @@ tools instead. move_to({"xyz": [-0.20, 0.10, 0.65], "gripper": -1, "tol": 0.008, "step_clip": 0.010, "max_steps": 80}) -# Then read states.json entry NN for final_eef_pos & final_dist_m. +# Inspect the returned log for final_eef_pos and final_dist_m. ``` diff --git a/robots/libero/guides/pro_hybrid_guide.md b/robots/libero/guides/pro_hybrid_guide.md index d7999405..500ff22f 100644 --- a/robots/libero/guides/pro_hybrid_guide.md +++ b/robots/libero/guides/pro_hybrid_guide.md @@ -13,14 +13,16 @@ This document layers on the base playbook. Read it first: - [`strict_hybrid_guide.md`](./strict_hybrid_guide.md) — the perception protocol: back-projection localization, the perception artifacts - (`images_cam/image_cam_NN.png` / `depths/depth_NN.npy` / `camera_meta.json`), + (`agentview_high.png`, `agentview_world_high.npy`, and + `agentview_metadata.json`), the primitive vocabulary, the Rules (0/1/2/4/5), and the `strict_perception` audit format. **This is the source of truth for *how you localize*.** - **This file** — LIBERO-Pro–specific setup, the four perturbation axes, the Pi0 fullshot baseline, the frame split, and how perception isolation changes the P2-swap story. -> Your task comes from `states.json[0].task_language`; the BDDL is FORBIDDEN. +> Your task comes from the initial `view_env_state` result's `task_language`; +> the BDDL is FORBIDDEN. > Read the authoritative instruction (the BDDL `:language` tag, coord-free) that > the runtime injects and obey it verbatim. Do **not** scrape the BDDL or import > the benchmark: that is error-prone (wrong task-map index → wrong task) and a @@ -42,19 +44,19 @@ recover in place or write an honest failure audit and `finish`. | | oracle mode (not in this repo) | **perception (this guide)** | |---|---|---| | how launched | oracle-state run | `rpent/cli/main.py --libero-type pro` (or `LIBERO_TYPE=pro`); perception artifacts always dumped, coords withheld | -| `states.json` objects | full `objects:{name:[x,y,z]}` | **`object_names:[…]` only — NO coords** | -| how you learn the task | env prompt / scrape BDDL | **`states.json[0].task_language`** (authoritative `:language`, coord-free) — never read the BDDL | -| extra obs artifacts | agentview RGB only | **+ `images_cam/`, `depths/`, `world/` (agentview); `images_wrist/`, `depths_wrist/`, `world_wrist/` (wrist); hi-res pairs; `camera_meta.json`** — all via `back_project` | +| returned `state` objects | full `objects:{name:[x,y,z]}` | **`object_names:[…]` only — NO coords** | +| how you learn the task | env prompt / scrape BDDL | **initial `task_language` returned by `view_env_state`** (authoritative `:language`, coord-free) — never read the BDDL | +| extra obs artifacts | agentview RGB only | **logical agentview/wrist image, depth, world-map, and metadata keys, including `agentview_high.png` and `wrist_high.png`** — geometry consumed through `back_project` | | cameras | agentview only | **agentview (fixed, ~1m → ±8-13cm) + eye-in-hand wrist (moves with gripper, ±1-2cm when <20cm to target)** — see §3.6 | -| how P2 swap is solved | read swapped coords from `states.json[0]` | **localize the swapped objects by `back_project`** | +| how P2 swap is solved | read swapped coords from oracle state | **localize the swapped objects by `back_project`** | | how a swapped *fixture* site is found | read the swap BDDL `:init` block | **localize the fixture visually** (see §3.5) | | run budget | short | **larger** — perception localization + manipulation is slower (raise `--max-turns` / `--planner-timeout-s`) | | audit `regime` | `strict` | `strict_perception` | -| which image you pick pixels in | agentview RGB | `images_cam/image_cam_NN.png` (or hi-res) for **pixel-picking**; `images/image_NN.png` only for a sanity glance | +| which image you pick pixels in | agentview RGB | `agentview_high.png` (or `agentview.png` at low resolution) for **pixel-picking**; never use `agentview_policy.png` for back-projection | **The single most important conceptual shift.** In oracle PRO mode the headline is "the hybrid beats Pi0 on P2 because it reads the *swapped* coordinates straight -out of `states.json[0]`, while Pi0 is prompt-/memory-blind." In **perception** +out of oracle state, while Pi0 is prompt-/memory-blind." In **perception** mode there are no coordinates to read — so the hybrid's P2 win now comes from **seeing where the object is and back-projecting it**. This is a *stronger* claim (no oracle state at all), and it is the whole point of running PRO in perception @@ -186,8 +188,8 @@ PRO scenes use one of three table fixtures; the eef home z differs by up to | `kitchen_table` | ≈ 1.17 | ≈ 0.90 | `(x∈±0.30, y∈±0.30)` | stove / cabinet / drawer / microwave | | `object` (low table) | ≈ 0.26 | ≈ 0.0 | `(x∈±0.30, y∈±0.30)` | `libero_object` grocery-into-basket | -**Mandatory check at session start: read `states.json[0].state.robot0_eef_pos[2]`** -(via `view_driver_state({"step": 0})`). ≈ 0.68 → LIVING_ROOM; ≈ 1.17 → KITCHEN; +**Mandatory check at session start: read `state.robot0_eef_pos[2]` from +`view_env_state({"step": 0})`.** ≈ 0.68 → LIVING_ROOM; ≈ 1.17 → KITCHEN; ≈ 0.26 → OBJECT. Pick `pre_pos_z` / `carry_z` / `release_z` accordingly (per-item OBJECT-frame altitudes are in `resources/libero/memory/project_libero_object_pro_done.md`). Sending a @@ -195,7 +197,7 @@ wrong-frame z (e.g. KITCHEN coordinates while the env is in LIVING_ROOM frame) crashes the env worker (EOFError, silent state loss). > **Perception note.** This proprioceptive z is *not* an object coordinate — it's -> the robot's own pose, which `states.json` still gives you in perception mode. +> the robot's own pose, which the returned `state` still gives you in perception mode. > Reading it to pick the frame is fine. It also doubles as a free depth sanity > check: your back-projected table z should sit ≈0.25 m below eef home z (≈0.43 in > LIVING_ROOM, ≈0.90 in KITCHEN). If a back-projection lands far from that, you @@ -228,15 +230,15 @@ slip in the gripper and the object ends up centimetres off target. Mitigation - `carry_z = 1.15` (higher than usual 1.10) - `step_clip = 0.020` (slower) - **Re-localize mid-travel** instead of trusting a cached `object_xyz - eef_xyz` - offset: `Read images_cam/image_cam_NN.png` (or the hi-res `images_cam_hi/`) - mid-carry and call `back_project` on the carried object's pixel; if the + offset: inspect `agentview_high.png` mid-carry and call `back_project` on the + carried object's pixel; if the perceived offset drifted >5 mm from post-pick, re-pre-position and re-`pi0_pick`. ### 3.4. Task language is from BDDL, not filename After the patch, `get_task(i).language` returns the perturbed `:language` tag, and -the env passes it to Pi0 as the prompt (surfaced to you as -`states.json[0].task_language`). For `_task` and `_lan` this is the perturbed +the env passes it to Pi0 as the prompt (surfaced as top-level `task_language` +in each returned state view). For `_task` and `_lan` this is the perturbed instruction. **Don't override it** — falsifying the VLA's prompt-blindness is the point. You read the same language to decide *which* object to localize and place. @@ -246,7 +248,7 @@ This is the biggest perception-mode trap, and it is **specific to `_swap`**. The P2 perturbation does not only swap loose objects; for `libero_goal_swap` it swaps entire **fixtures** (stove ↔ cabinet ↔ wine_rack), so a goal predicate like `On(bowl, flat_stove_1_cook_region)` now points at wherever the *stove* was -relocated to, and there are no coordinates in `states.json` to read. See +relocated to, and there are no coordinates in the returned `state`. See `resources/libero/memory/feedback_swap_perturbs_fixtures.md`. In **oracle** mode the documented fix is to read the swap BDDL `:init` block and @@ -256,7 +258,7 @@ recompute the fixture site's world coordinates. **That is forbidden here** — t 1. Identify the target fixture by name from the (perturbed) task language and `state.object_names`. 2. **Localize the fixture's predicate site visually** in - `images_cam_hi/image_cam_hi_NN.png`: pick pixels on the stove's cook region / + `agentview_high.png`: pick pixels on the stove's cook region / cabinet top surface / rack top shelf, then `back_project` (sample 3–5 pixels, median the xy — fixtures are large and the surface you want is the *placement* surface, not the nearest edge). @@ -272,9 +274,9 @@ where only bowls/plates move) are simpler: just localize each object by ### 3.6. Two cameras — agentview = IDENTITY, wrist = GEOMETRY Do **not** restate the two-camera protocol here — the strict guide's **First-step -perception protocol** is the source of truth: agentview / agentview-hi is the +perception protocol** is the source of truth: `agentview.png` / `agentview_high.png` is the semantic identity authority (decides *which* object/surface satisfies the task -language + relation); wrist / wrist-hi refines geometry for the *same* candidate +language + relation); `wrist.png` / `wrist_high.png` refines geometry for the *same* candidate (accept only within ~3–5 cm of the agentview anchor, never average, basket/cavity excepted). Both maps share one world frame, read via `back_project({"camera":"wrist"})`; the optional `segment` honours both cameras @@ -352,7 +354,7 @@ seed-0 reference corpus**, not a write target. There is no manual server to launch and no REPL to drive: the MCP runner starts, manages, and tears down `env_server.py` for you and blocks each tool call until -the next `states.json` entry is dumped. Do not start, stop, or background it, and +the next state record is dumped. Do not start, stop, or background it, and do not poll for readiness. ### 4.3. Pi0 fullshot baseline @@ -366,7 +368,7 @@ Pi0 never sees object coords in either mode, so there is no perception variant o the baseline. Expected behavior: - **P1 (task):** Pi0 "succeeds at the wrong task" — it picks the *base*-task - target object, places it on the plate, and `libero_terminated=False` because the + target object, places it on the plate, and `terminated=False` because the goal predicate names a different object. This is exactly the gap the hybrid closes. - **P2 (position):** Pi0 picks / places at the *base* (un-swapped) location. @@ -383,20 +385,20 @@ add the PRO fields: "seed": 0, "regime": "strict_perception", "perturbation_type": "swap (P2 Position perturbation)", - "perturbed_task_language": "", + "perturbed_task_language": "", "perturbation_semantics": "", "expected_baseline_behavior": "", - "strategy_notes": "HOW you localized — which pixel(s) in images_cam, back-projected world xyz; for swap, how you found the relocated object/fixture", + "strategy_notes": "HOW you localized — which pixel(s) in agentview_high.png, back-projected world xyz; for swap, how you found the relocated object/fixture", "pick_result": { /* the pi0_pick step's result */ }, - "final_state": { /* latest states.json entry's `state` field */ }, - "libero_terminated": true + "final_state": { /* latest view_env_state result's `state` field */ }, + "terminated": true } ``` `strategy_notes` **must** describe the localization (pixel → `back_project` → world xyz). For swap cells, explicitly note that the relocated object/fixture was found by perception, not by reading coords. If unrecoverable after honest -exploration, write `libero_terminated: false` with what you tried and which step +exploration, write `terminated: false` with what you tried and which step failed — never warp (teleport primitives are deleted; see Rule 4 in the perception protocol). Write the audit with `write_text_file` to `{output_dir}/{recipe_tag}.json`, then call `finish`. @@ -405,8 +407,8 @@ perception protocol). Write the audit with `write_text_file` to - **Rule 0 (use images for reasoning).** Even more critical under PRO: P2 swap can move a large object/fixture clear across the table, and you have *no* - coordinates to fall back on. `images_cam/image_cam_NN.png` + depth are your only - spatial truth — open `images_cam/` and describe the scene before deciding + coordinates to fall back on. `agentview_high.png` plus `back_project` are your + spatial truth — inspect the embedded image and describe the scene before deciding targets. - **Rule 1 (no `pi0_end_to_end`).** Pi0 does the grasp via `pi0_pick`; the LLM scripts every motion + release. Under PRO this is doubly important — handing @@ -419,7 +421,7 @@ perception protocol). Write the audit with `write_text_file` to `finish`. `_swap` typically needs an in-episode retry — document it. - **Rule 4 (no teleport).** `set_object_pose`, `articulate_to`, `js_move_to`, `carry_object` are deleted. A goal past OSC reach with no physical approach → - honest `libero_terminated:false`. + honest `terminated:false`. - **Rule 5 (assume solvable).** A localization that moves the gripper into thin air means you picked a wrong pixel (a reflection, a rim, a decoy object under the Object perturbation), not that the cell is unreachable. Re-look, re-pick, @@ -461,7 +463,7 @@ coordinates (re-derive every xyz from THIS scene) and never write there. scene; a perception recipe is *data-flow* (perceive → plan → act), never hard-coded xyz. 3. **Replicate on `libero_object`, `libero_goal`, `libero_10`.** Frame split - applies (read `states.json[0].state.robot0_eef_pos[2]`). For + applies (read the initial returned `state.robot0_eef_pos[2]`). For `libero_goal_swap`, apply §3.5 — localize the **swapped fixture** visually. 4. **Aggregate into a main table** `(suite × perturbation × {Pi0, hybrid})`. The headline number is the conditional: of the seeds Pi0 fails on, what fraction @@ -486,8 +488,8 @@ python rpent/cli/main.py --env libero --suite libero_spatial_swap --task --s Then, inside the run: -4. `view_driver_state({"step": 0})` → read `state.robot0_eef_pos[2]` to pick the - frame (§3.1). `Read images_cam/image_cam_00.png` (or hi-res) and +4. `view_env_state({"step": 0})` → read `state.robot0_eef_pos[2]` to pick the + frame (§3.1). Inspect `agentview_high.png` and call `view_camera_meta`. Localize the target (and, for `_swap`, the relocated object/fixture) with `back_project` — run the mandatory pre-task perception pass (§3.6c). Plan, then execute one structured tool at a time. diff --git a/robots/libero/guides/strict_hybrid_guide.md b/robots/libero/guides/strict_hybrid_guide.md index 683feabe..2cc9d5e6 100644 --- a/robots/libero/guides/strict_hybrid_guide.md +++ b/robots/libero/guides/strict_hybrid_guide.md @@ -13,10 +13,10 @@ state JSON carried GT object coordinates, is not included here). | | legacy oracle mode (not in this repo) | **perception (this guide)** | |---|---|---| -| how object coords are withheld | (none — full GT coords in `state`) | **the runner withholds them: `states.json` carries `object_names` only, no coordinates** | -| `states.json` objects | full `objects:{name:[x,y,z]}` | **`object_names:[…]` only — NO coords** | -| how you learn the task | env prompt / BDDL | **`states.json[NN].task_language`** (authoritative `:language`, coord-free) — never scrape the BDDL | -| extra obs artifacts | `images/image_NN.png` only | **+ `images_cam/`, `depths/`, `world/` (agentview); `images_wrist/`, `depths_wrist/`, `world_wrist/`, `wrist_meta/` (wrist); top-level `camera_meta.json`; hi-res `images_cam_hi/`, `world_hi/`, `images_wrist_hi/`, `world_wrist_hi/`** | +| how object coords are withheld | (none — full GT coords in `state`) | **the returned `state` carries `object_names` only, no coordinates** | +| returned `state` objects | full `objects:{name:[x,y,z]}` | **`object_names:[…]` only — NO coords** | +| how you learn the task | env prompt / BDDL | **top-level `task_language` returned by `view_env_state`** (authoritative `:language`, coord-free) — never scrape the BDDL | +| extra obs artifacts | policy image only | **logical agentview/wrist image, depth, world-map, and metadata keys, including `agentview_high.png` and `wrist_high.png`** | | cameras | agentview only | **agentview (fixed, ~1m → ±8–13 cm) + eye-in-hand wrist (moves with gripper, ±1–2 cm when <20 cm to target)** — coarse→fine, see below | | how you get an object's xyz | read `state["objects"][name]` | **pick the object pixel, then `back_project({"row":ROW,"col":COL,"step":NN})` (K⁻¹+extrinsic already done); refine with the wrist map up close** | | how you confirm the grasp | GT object-lift oracle | **grasp-only, no oracle — you judge the grasp from gripper width + wrist cam (Rule 1 / 1b)** | @@ -26,22 +26,16 @@ state JSON carried GT object coordinates, is not included here). How localization works (the core of this mode): -The toolkit already back-projects EVERY pixel for you into world coordinates and -saves them as `world/world_NN.npy` (agentview) and `world_wrist/world_wrist_NN.npy` -(wrist) — both in the SAME world frame. **You do NOT write back-projection math; -you pick a pixel and call `back_project`, which indexes the map for you.** +The toolkit already back-projects EVERY pixel for you through +`agentview_world.npy` / `agentview_world_high.npy` or +`wrist_world.npy` / `wrist_world_high.npy` — all in the SAME world frame. +These are logical artifact names scoped by step, not paths to construct. +**You do NOT write back-projection math; you pick a pixel and call +`back_project`, which selects the matching map for you.** -> **Path convention:** below, a bare file name refers to the file inside its own -> `output_dir` subdirectory — e.g. `image_cam_hi_NN.png` lives at -> `images_cam_hi/image_cam_hi_NN.png`, `world_NN.npy` at `world/world_NN.npy`, -> `image_wrist_hi_NN.png` at `images_wrist_hi/image_wrist_hi_NN.png` (see the obs -> artifacts row above for the full directory list). `NN` is the zero-padded step -> index. In practice you rarely hand-build these paths — each primitive tool -> already returns the resolved `image_cam_hi_path` etc. for the new step. - -1. Read `image_cam_hi_NN.png` — agentview RGB **in the calibration frame** +1. Inspect `agentview_high.png` — agentview RGB **in the calibration frame** (vertical-flip of the raw buffer). This is the image you pick pixels in. - `image_NN.png` is the Pi0-rotation frame; **do not pick pixels there**. + `agentview_policy.png` is the Pi0 frame; **do not pick pixels there**. 2. Find the target object visually → pixel `(row, col)`. 3. Read its world xyz directly: `back_project({"row":ROW,"col":COL,"step":NN})`. Sample 3–5 pixels on the object's top surface and median the returned xy @@ -50,10 +44,9 @@ you pick a pixel and call `back_project`, which indexes the map for you.** ### Hi-res perception channel (ON BY DEFAULT, 1024×1024) -The toolkit dumps, every step, a 1024×1024 pair per camera IN ADDITION to the -256 files: `images_cam_hi/image_cam_hi_NN.png` + `world_hi/world_hi_NN.npy` -(agentview) and `images_wrist_hi/image_wrist_hi_NN.png` + -`world_wrist_hi/world_wrist_hi_NN.npy` (wrist). +The toolkit records, every step, a 1024×1024 pair per camera IN ADDITION to the +256 artifacts: `agentview_high.png` + `agentview_world_high.npy` and +`wrist_high.png` + `wrist_world_high.npy`. **PREFER the hi pair for looking and identification** — a far object spans 4× more pixels, and package text/label art is actually legible (e.g. "Cream Cheese" vs "BUTTER" boxes are readable at 1024 and indistinguishable smears @@ -69,20 +62,19 @@ at 256). If the hi files are absent, everything below works unchanged at 256. (the residual ~2 cm error is the surface-vs-center offset, not pixel size) — the hi channel is for **identification**, not for replacing the wrist-cam fine-localization protocol. -- Disk note: hi files keep only the LAST ~5 steps (rolling window); the 256 - history is complete. If the hi files are absent, everything below works - unchanged at 256. +- If a high-resolution artifact is absent, everything below works unchanged + with `agentview.png` or `wrist.png` and `resolution:"low"`. ### Two cameras — agentview = IDENTITY, wrist = GEOMETRY **Roles are NOT symmetric (this is the core discipline, from the 80-task localization sweep):** -- **agentview** (`image_cam_hi_NN.png` + `world_hi/world_hi_NN.npy`, ~1 m away): +- **agentview** (`agentview_high.png` + `agentview_world_high.npy`, ~1 m away): the **semantic IDENTITY authority** — decides WHICH object/surface satisfies the task language + spatial relation. At 1024 a far object spans enough pixels to read labels/shape. Its metric precision is only ±8–13 cm, but identity is its job, not millimetres. -- **wrist** (`image_wrist_hi_NN.png` + `world_wrist_hi/world_wrist_hi_NN.npy`, +- **wrist** (`wrist_high.png` + `wrist_world_high.npy`, <20 cm → ±1–2 cm): a **GEOMETRY refinement** camera for the SAME candidate agentview already chose. It is near-vertical and **bad at identity** — it cannot read side labels or tell duplicate/similar items apart, and in failed @@ -90,13 +82,13 @@ localization sweep):** wrist freely re-identify a non-basket target.** Protocol (non-basket objects/surfaces): -1. **Identity (agentview)** — in `image_cam_hi_NN.png` choose the target by RGB / +1. **Identity (agentview)** — in `agentview_high.png` choose the target by RGB / label / shape / global spatial relation; sample 3–8 pixels on it and median `back_project` → the **identity anchor** (rough xy ok). 2. **Approach** — `move_to` ~15–20 cm directly above that anchor xy (toolkit re-renders both cams after every primitive). 3. **Geometry refine (wrist)** — pick the SAME candidate's pixel in - `image_wrist_hi_NN.png`, `back_project` it with `camera:"wrist"`. **Accept + `wrist_high.png`, `back_project` it with `camera:"wrist"`. **Accept this corrected xy ONLY if it is within ~3–5 cm of the agentview anchor**; if it jumps >5 cm, REJECT it (it hit a look-alike/background) and keep the agentview xyz. The wrist may sharpen coordinates but may NOT override the @@ -124,7 +116,8 @@ entities up front instead of recovering later. > (row−cy)·z/fy, z]` with `z=depth[row,col]`, then `P_world = > extrinsic_cam2world @ [cam,1]`. You must invert `K` BEFORE the extrinsic — the > old `E @ [col·z, row·z, z, 1]` recipe (no `K⁻¹`) is **wrong** (metres off). -> The `world/` and `world_wrist/` maps bake in the correct `K⁻¹`+extrinsic, and +> The agentview and wrist world-map artifacts bake in the correct +> `K⁻¹`+extrinsic, and > `back_project` reads them for you — that is precisely why `back_project` > exists; you never write this math yourself. Forward world→pixel was verified > 5/5 vs GT at design time (plate Δ=6 mm). The wrist map may be all-table/null @@ -170,10 +163,10 @@ place — YOU do every `move_to` and the `release`. After a pick, decide "did I grab the target?" from two coord-free signals: -- **Gripper** (`state.robot0_gripper_qpos` from the latest `states.json` - entry): fingers closed but NOT fully shut (~0.01–0.05 gap) ⇒ holding an +- **Gripper** (`state.robot0_gripper_qpos` from the latest returned state + record): fingers closed but NOT fully shut (~0.01–0.05 gap) ⇒ holding an object; fully closed (~0.0) ⇒ grasped air. -- **Wrist cam**: after the lift, read `image_wrist_hi_NN.png` — the target +- **Wrist cam**: after the lift, inspect `wrist_high.png` — the target should be raised into the gripper and the spot it came from now empty (its surface z jumps up by your lift distance). If needed, `back_project` wrist pixels to confirm the surface z jumped. @@ -181,29 +174,28 @@ After a pick, decide "did I grab the target?" from two coord-free signals: `pi0_pick`'s returned `success` (an eef-lift + gripper-closure heuristic, pure proprioception) is a HINT, not proof — confirm with the wrist cam before carrying. The only authoritative TASK-success signal is -`state.libero_terminated` (the benchmark predicate), which is name-independent. +top-level `terminated` (the benchmark predicate), which is name-independent. ## Rule 4 — NO TELEPORT primitives (physics-only) `set_object_pose`, `articulate_to`, `js_move_to`, `carry_object` are **deleted from the codebase**. They are not callable and are not in the tool list. If a goal is past OSC reach and no physical approach works, write an honest -`libero_terminated:false` audit — never warp. +`terminated:false` audit — never warp. ## Rule 0 — Use images for reasoning, not just JSON state -After every primitive tool call, inspect the returned state and image paths -(the tool returns them — no separate read needed). Read the new -`image_cam_hi_NN.png` path (calibration frame — the one you pick pixels in) -and, when close to a target, the `image_wrist_hi_NN.png` path. Even more +After every primitive tool call, inspect the returned state and embedded images +(the tool returns them — no separate read needed). Inspect +`agentview_high.png` (calibration frame — the one you pick pixels in) +and, when close to a target, `wrist_high.png`. Even more important here than in oracle mode: this is *the* signal you use to find objects. -`image_cam_hi_NN.png` (and its 256 counterpart `image_cam_NN.png`) is the -calibration-frame RGB — same scene as `image_NN.png` but vertically flipped so -that pixel coordinates align with the camera matrices in `camera_meta.json`. -Pick object pixels from these; `states.json` alone gives only proprioception + -object names. +`agentview_high.png` (and its 256 counterpart `agentview.png`) is the +calibration-frame RGB — the same scene as `agentview_policy.png`, oriented so +that pixel coordinates align with `agentview_metadata.json`. Pick object pixels +from these; the returned JSON `state` gives only proprioception + object names. ## Rule 2 — SINGLE EPISODE, NO RESET @@ -213,20 +205,20 @@ and do NOT restart the episode. You MAY recover *within* this one episode grasp; walk the next rung of the Pi0 prompt ladder in Rule 3; re-firm the grip; `rotate_pitch`/`move_pose`) — that is all one continuous attempt. But the instant you would want to start over, **STOP instead and write the audit** -(success or an honest `libero_terminated:false`), then call `finish`. Never +(success or an honest `terminated:false`), then call `finish`. Never warp; never reset. ## Rule 5 — Assume every task is physically solvable Same as oracle mode. A localization that "looks right" but moves the gripper into thin air usually means your pixel was on a wrong surface (e.g. picked the -bowl's reflection on the table). Re-look at `image_cam_hi_NN.png`, pick a +bowl's reflection on the table). Re-look at `agentview_high.png`, pick a different pixel firmly on the target's top, re-`back_project`. Don't conclude "unreachable" until you've validated localization. ## Rule 6 — Your task is `task_language`; the BDDL is FORBIDDEN -Each `states.json[NN]` entry carries a **`task_language`** field — the +Each state view carries a top-level **`task_language`** field — the authoritative task instruction (the BDDL's `:language` tag, which contains **no** coordinates). **Read it and obey it verbatim.** Do not infer the task from object names, from sibling recipes, or by guessing a `task_map` index — @@ -264,11 +256,10 @@ Pick the target purely from where things ARE. (You never need to know which stop, or restart it. 2. **You call one structured MCP tool per step.** The tool BLOCKS until the toolkit runs that one primitive and dumps the new step, then RETURNS the new - state + log + image paths. There is no file bus and no polling — the tool's - return value IS your signal. Each dumped step appends an entry to - `states.json` (entry `[NN]`) and writes `images/image_NN.png` + - `images_cam/image_cam_NN.png` + `depths/depth_NN.npy` + `world/world_NN.npy` - (+ the wrist and hi-res dirs) and, once, top-level `camera_meta.json`. + state + log + embedded images. There is no file bus and no polling — the + tool's return value IS your signal. Each dumped step records logical artifact + names such as `agentview_high.png`, `agentview_world_high.npy`, + `wrist_high.png`, and `wrist_world_high.npy` in that step's `artifacts` list. 3. **You read the returned state, localize via `back_project`, decide the next move, and call the next tool.** @@ -276,30 +267,28 @@ Pick the target purely from where things ARE. (You never need to know which The runner (`rpent/cli/main.py`) launches and owns the env server (Pi0.5 + single-env sim) — do not start/stop it. You call MCP tools; begin by reading step 0 via -`view_driver_state({"step":0})`. +`view_env_state({"step": 0})`. ## The perception artifacts you read each step -| artifact | what's in it | +| logical artifact or field | what's in it | |---|---| -| `states.json` (entry `[NN]`) | `step_idx`, `libero_terminated`, **`task_language` (your authoritative task instruction — the BDDL `:language` tag, coord-free; obey it verbatim)**, `state.{robot0_eef_pos, robot0_eef_quat, robot0_gripper_qpos, object_names}`, and the merged `command`/`result`/`elapsed_s` for that step. **No object coordinates.** Read via `view_driver_state({"step": NN})` (omit `step` = latest). | -| `images/image_NN.png` | RGB in Pi0 frame (180° rotated). *Do not pick pixels here for back-projection.* | -| `images_cam/image_cam_NN.png` | agentview RGB in **calibration frame** (vertical flip). Pick object pixels HERE (256 grid → `back_project` with `resolution:"low"`). | -| `images_cam_hi/image_cam_hi_NN.png` | **HI-RES 1024×1024** agentview RGB, calibration frame. **PREFER this for looking / identification**; `back_project` its pixels at the default `resolution:"high"`. | -| `depths/depth_NN.npy` | `(256, 256) float32` agentview metric depth (m), calibration frame. Same row/col as `image_cam_NN.png`. | -| `world/world_NN.npy` | `(256, 256, 3) float32` — **precomputed agentview world xyz per pixel** (K⁻¹+extrinsic done). Prefer `back_project`; read this manually only for debugging. Fixed cam (~1 m). | -| `world_hi/world_hi_NN.npy` | `(1024, 1024, 3) float16` — precomputed agentview world xyz per hi-res pixel. | -| `images_wrist/image_wrist_NN.png` (+ `images_wrist_hi/`) | **wrist (eye-in-hand) RGB**, calibration frame. Moves with the gripper. | -| `depths_wrist/depth_wrist_NN.npy` | wrist metric depth (m). | -| `world_wrist/world_wrist_NN.npy` (+ `world_wrist_hi/`) | `(256, 256, 3)` / `(1024, 1024, 3)` **precomputed wrist world xyz per pixel**, SAME world frame as agentview. ±1–2 cm when <20 cm to target. May be all-table/null until you move over the target. | -| `wrist_meta/wrist_meta_NN.json` | wrist intrinsics + extrinsic **for THAT step only** (the wrist cam moves). Read via `view_camera_meta({"camera":"wrist","step":NN})`. | -| `camera_meta.json` (top-level) | agentview `intrinsic_K` (3×3), `extrinsic_cam2world` (4×4), `depth_near/far`, projection recipe. Read via `view_camera_meta({"camera":"agentview"})`. | -| `segments/segment_NN_XX.json` | (after a completed `segment` response) `{found, mode, prompt|point, camera, source_step, score, box, mask_shape, centroid_pixel, world_xyz, n_pixels}` — SAM3's top mask back-projected via the matching world map. `world_xyz` is a robust MEDIAN over the whole mask. A valid no-detection response carries `{found:false, error}`. `XX` is a per-step index. | -| `segments/segment_overlay_NN_XX.png` | (only after a successful `segment` call) the segmented mask tinted red on the source image — read it to confirm SAM3 grabbed the right object. | - -The command + its result + `elapsed_s` are merged INTO the `states.json[NN]` -entry (and echoed in each primitive tool's return value) — there is no separate -per-step log file to read. +| `view_env_state` result | `step`, top-level `task_language`, `terminated`, `truncated`, `state.{robot0_eef_pos, robot0_eef_quat, robot0_gripper_qpos, object_names}`, `artifacts`, and nested `log.{command,result,elapsed_s}`. **No object coordinates.** Step `-1` selects latest. | +| `agentview_policy.png` | RGB in Pi0 frame. *Do not pick pixels here for back-projection.* | +| `agentview.png` | 256×256 agentview RGB in **calibration frame**. Pick object pixels HERE only with `back_project(..., resolution:"low")`. | +| `agentview_high.png` | **HI-RES 1024×1024** agentview RGB, calibration frame. **PREFER this for looking / identification**; `back_project` defaults to `resolution:"high"`. | +| `agentview_depth.npy` | 256×256 agentview metric depth (m), aligned with `agentview.png`. Consume through `back_project`. | +| `agentview_world.npy` / `agentview_world_high.npy` | Precomputed agentview world xyz per low/high-resolution pixel. Prefer `back_project`; do not open them manually. | +| `wrist.png` / `wrist_high.png` | **Wrist (eye-in-hand) RGB**, calibration frame. Moves with the gripper. | +| `wrist_depth.npy` | Wrist metric depth (m), aligned with `wrist.png`. | +| `wrist_world.npy` / `wrist_world_high.npy` | Precomputed wrist world xyz per low/high-resolution pixel in the SAME world frame as agentview. May be all-table until you move over the target. | +| `wrist_metadata.json` | Wrist intrinsics + extrinsic **for THAT step only**. Read via `view_camera_meta({"camera":"wrist","step":NN})`. | +| `agentview_metadata.json` | Agentview intrinsics, cam-to-world extrinsic, depth range, and projection notes. Read via `view_camera_meta({"camera":"agentview","step":NN})`. | +| `segment_XX.json` | Per-step segment record named by the returned `segment_artifact`; includes mode, camera, source step, score, box, centroid, and `world_xyz`. | +| `segment_overlay_XX.png` | Per-step overlay named by `overlay_artifact` and embedded by a successful `segment` result for visual confirmation. | + +The command, result, and `elapsed_s` are returned under `log`; there is no +separate per-step log file to read. ## Localization with `back_project` (coarse agentview → fine wrist) @@ -311,7 +300,7 @@ You index the precomputed map through `back_project` — no K⁻¹ math. Coarse back_project({"row": ROW, "col": COL, "step": NN}) # FINE: wrist — after move_to ~15-20cm above the target, pick its pixel in -# image_wrist_hi_NN.png and back_project it (±1-2cm; refines the SAME candidate). +# wrist_high.png and back_project it (±1-2cm; refines the SAME candidate). back_project({"row": ROW, "col": COL, "step": NN, "camera": "wrist"}) ``` @@ -335,7 +324,7 @@ Tips: Call one structured tool per step. The full primitive set (only the *control signals* are listed here) — every one blocks until the step is dumped and -returns the new state + log + image paths: +returns the new state, log, artifact names, and embedded images: ```jsonc // === physics-only primitives (the entire allowed set) ===================== @@ -352,7 +341,7 @@ pi0_pick({"prompt": "pick up the X", "max_chunks": 20, "lift_thresh": 0.05, "gripper_closed_thresh": 0.06}) // Pi0.5 for a contact skill (knob turn, drawer/door open-close — rare here). -// success mirrors libero_terminated only; inspect image/state for intermediates. +// success mirrors terminated only; inspect image/state for intermediates. pi0_doubled({"prompt": "turn off the stove", "max_chunks": 20}) // Open gripper to place. Triggers libero termination if the On/In predicate met. @@ -379,15 +368,15 @@ move_pose({"xyz": [x, y, z], "target_pitch": 0.0, "target_yaw": 0.0, // SAM3-grounded localization — does NOT move the robot and does NOT // replace manual back-projection. Segments the most-recent dumped image for the // prompt, back-projects the mask through the matching world map, and writes -// segments/segment_NN_XX.json {score, box, centroid_pixel, world_xyz (robust MEDIAN over -// the whole mask), n_pixels} + segments/segment_overlay_NN_XX.png. Read world_xyz +// segment_XX.json {score, box, centroid_pixel, world_xyz (robust MEDIAN over +// the whole mask), n_pixels} + segment_overlay_XX.png. Read world_xyz // directly instead of eyeballing a pixel. camera "wrist" uses the wrist world // map for fine refinement (move the eef over the target first). Pass // "point":[row,col] for a point prompt instead of text; prompt and point are // mutually exclusive. min_score default 0.2. // RPent manages the SAM3 service. If one call fails or finds nothing, the result is // an {"error":...,"fallback":...} dict — fall back to picking a pixel in -// image_cam_hi_NN.png and calling back_project. +// agentview_high.png and calling back_project. segment({"prompt": "the black bowl on the stove", "camera": "agentview", "point": null, "min_score": 0.2}) @@ -401,9 +390,9 @@ segment({"prompt": "the black bowl on the stove", "camera": "agentview", that is a robust MEDIAN over the whole object mask (hundreds of pixels), which beats eyeballing 3–5 pixels. Workflow: -1. Read `image_cam_hi_NN.png`, decide the target by the task's spatial RELATION. +1. Inspect `agentview_high.png`, decide the target by the task's spatial RELATION. 2. Call `segment` with a **plain visual phrase + the relation**, then read the - returned `world_xyz` and the `segments/segment_overlay_NN_XX.png` to confirm the mask + returned `world_xyz` and embedded overlay image to confirm the mask is on the right object before you move. 3. **Two camera views — YOUR choice via the `"camera"` field** (`segment` works on either; default `"agentview"`): @@ -438,10 +427,10 @@ beats eyeballing 3–5 pixels. Workflow: A typical bowl→plate cell looks like: -1. `view_driver_state({"step":0})`; inspect `task_language`, - `image_cam_hi_00.png`, and `camera_meta.json`. +1. `view_env_state({"step": 0})`; inspect `task_language`, + `agentview_high.png`, and call `view_camera_meta` if needed. 2. Identify the target by the task's spatial RELATION, not its name (Rule 7). -3. Localize it — pick its pixel in `image_cam_hi_00.png`, +3. Localize it — pick its pixel in `agentview_high.png`, `back_project({"row":ROW,"col":COL,"step":0})` (coarse). Refine with the wrist cam after pre-positioning (fine). 4. **Pre-position** ~15–20 cm above it: `move_to([obj_x, obj_y, carry_z])`, @@ -453,7 +442,7 @@ A typical bowl→plate cell looks like: 7. Localize the placement region (basket / plate / drawer slot) the same way. 8. `move_to([place_x, place_y, carry_z])` to traverse at constant height. 9. Optionally descend `move_to([place_x, place_y, place_z])`. -10. `release` — predicate (`On`/`In`) checks → `libero_terminated=True` if hit. +10. `release` — predicate (`On`/`In`) checks → `terminated=True` if hit. 11. Light retreat (`move_to` upward) so the next step's image is clean. > **Predicate fire timing.** Most LIBERO `On(X, Y)` predicates fire on @@ -492,34 +481,38 @@ qualifier** for elevated picks (stove, cabinet-top, drawer). See ## Reading state After every primitive tool call, the return value already carries the new -`state`, the merged `command`/`result`/`elapsed_s` log, and the image paths — +`state`, nested `log.{command,result,elapsed_s}`, artifact names, and embedded +images — you do **not** need a separate read. When you need an older step, call -`view_driver_state({"step":NN})` (omit `step` for the latest): +`view_env_state({"step": NN})` (use `step: -1` for the latest): -1. Check `result.success`, `final_dist_m`, `peak_lift_m`, etc. in the returned - log. -2. Read the returned `image_cam_hi_NN.png` path → visual confirmation; pick +1. Check `log.result.success`, `log.result.final_dist_m`, + `log.result.peak_lift_m`, etc. +2. Inspect the embedded `agentview_high.png` image → visual confirmation; pick pixels for any new localization. -3. Check `state.robot0_eef_pos` / `robot0_gripper_qpos` / `libero_terminated` - in the returned `state`. +3. Check `state.robot0_eef_pos`, `state.robot0_gripper_qpos`, and top-level + `terminated` in the returned view. You **do not** open the depth maps yourself — feed a pixel to `back_project`. -Don't call `view_driver_state` immediately after a primitive that already +Don't call `view_env_state` immediately after a primitive that already returned the new state. ## Common failure modes -- **Pick missed (gripper closed empty).** `result.peak_lift_m` < `lift_thresh`, - `min_gripper_opening` ≈ 0. Re-pre-position 1–2 cm lower or shifted; retry +- **Pick missed (gripper closed empty).** `log.result.peak_lift_m` < + `lift_thresh`, `log.result.min_gripper_opening` ≈ 0. Re-pre-position 1–2 cm + lower or shifted; retry Pi0 with next prompt-ladder rung. -- **Object slipped mid-carry.** `release` returns `term=False` and the object +- **Object slipped mid-carry.** `release` returns top-level + `terminated=false` and the object is no longer where you `release`d it. `release`, re-pre-pos above it, `pi0_pick` again, traverse again. -- **OSC stuck.** `move_to` returns `final_dist_m > 0.05` at `max_steps` and +- **OSC stuck.** `move_to` returns `log.result.final_dist_m > 0.05` at + `log.result.max_steps` and same xy twice → try `rotate_pitch`, split into more waypoints, or `move_pose` (co-varying) for the cabinet-front singularity. - **Placement off because localization was wrong.** The release puts the - object on bare table instead of on the plate. Re-read `image_cam_hi_NN.png`, + object on bare table instead of on the plate. Re-inspect `agentview_high.png`, pick a different pixel on the target region (sample 3 pixels on the plate's flat top, median the back-projected xy), redo. Tip: if depth at your chosen pixel is much closer than the table z, you picked the camera's near edge / @@ -528,33 +521,32 @@ returned the new state. ## Verifying strict compliance Before saving the audit, confirm your command history is physics-only. The -teleport primitives are not even in the tool list, but audit it anyway: read -the `command` field of every `states.json` entry (via -`view_driver_state({"step":NN})` per step, or `read_text_file` on -`{output_dir}/states.json`) and confirm every `command.action` is one of the +teleport primitives are not even in the tool list, but audit it anyway: inspect +each recorded step with `view_env_state` and confirm every `log.command.action` +is one of the allowed physics primitives (`move_to`, `pi0_pick`, `pi0_doubled`, `release`, `set_gripper`, `rotate_wrist`, `rotate_pitch`, `move_pose`) — no `set_object_pose` / `articulate_to` / `js_move_to` / `carry_object` appears. ## Persisting successful runs as audit JSONs -When `state.libero_terminated == true`: +When top-level `terminated == true`: a. The working command recipe (`{output_dir}/recipe_{recipe_tag}.jsonl`) is - **auto-exported by the runner** from the non-error primitive commands in - `states.json` plus successful segment calls recorded in - `segments/segment_*.json`, merged in execution order — you do NOT + **auto-exported by the runner** from non-error primitive commands in the + recorded state trace plus successful `segment_XX.json` artifacts, merged in + execution order — you do NOT hand-write it. b. Write a minimal audit JSON with `write_text_file` to `{output_dir}/{recipe_tag}.json` with at least: `suite`, `task_id`, `seed`, `regime: "strict_perception"`, `strategy_notes` (mention HOW you localized — which pixel, depth, back-projected world xyz), `pick_result` (the `result` - from your `pi0_pick`), `final_state` (the latest `states.json` entry's - `state` field), `libero_terminated: true`. + from your `pi0_pick`), `final_state` (the latest returned `state` field), + `terminated: true`. c. Call `finish({"status":"success","summary":"…"})`. If unrecoverable after honest exploration in this one episode, write -`{output_dir}/{recipe_tag}.json` with `libero_terminated: false` + +`{output_dir}/{recipe_tag}.json` with `terminated: false` + `strategy_notes` describing what you tried, the back-projected xyz you used, and which step failed. Then call `finish` (NO reset, NO second attempt). @@ -564,7 +556,7 @@ which step failed. Then call `finish` (NO reset, NO second attempt). ## Iteration heuristics - After 2 failed retries on the same step, **stop tuning numerics** and - inspect images: read `image_cam_hi_NN.png` at pick / pre-release / + inspect `agentview_high.png` at pick / pre-release / post-release. The visual disagreement is usually the bug (you picked the wrong pixel, or Pi0 grabbed the decoy). This is the lesson of `feedback_failure_forensics.md` — applies even more strongly here. @@ -581,14 +573,14 @@ which step failed. Then call `finish` (NO reset, NO second attempt). - **No GT object coordinates anywhere in your reasoning.** The `state` you read has none; the only legitimate sources of object xyz are the camera - images + depth + the precomputed `world/` and `world_wrist/` maps (via + images + depth + the precomputed agentview and wrist world maps (via `back_project`). - **No teleport primitives.** The four are deleted. - **Pi0 only does the grasp.** You script every motion + release. - **Fully oracle-free, including the grasp.** There is no GT-lift oracle — `pi0_pick` reads NO GT object pose and takes no tracking argument. You judge the grasp from gripper width + the wrist cam, and TASK success from - `state.libero_terminated` (the benchmark predicate). + top-level `terminated` (the benchmark predicate). - **Single attempt.** One episode, no reset (Rule 2). - The expected audit `regime` is `strict_perception`. @@ -606,6 +598,6 @@ When you write a new audit, browse a sibling cell's `recipe_{tag}.jsonl` as a `back_project` from THIS scene's depth. Begin by reading `resources/libero/memory/MEMORY.md`, then call -`view_driver_state({"step":0})` and inspect the returned `image_cam_hi_00.png` -(+ `camera_meta.json` via `view_camera_meta`); localize the target object via +`view_env_state({"step": 0})` and inspect `agentview_high.png` +(+ metadata via `view_camera_meta`); localize the target object via `back_project`, then plan and execute. diff --git a/robots/libero/prompts/system.py b/robots/libero/prompts/system.py index 16bd1e93..657c4bfc 100644 --- a/robots/libero/prompts/system.py +++ b/robots/libero/prompts/system.py @@ -10,12 +10,13 @@ > persistence / up to N attempts" instruction anywhere below).** This is a > ONE-SHOT evaluation: you get **exactly ONE episode**. You MUST NOT call > `reset`, and you must not restart the episode. Plan carefully, then execute -> your single best manipulation sequence toward `state.libero_terminated == true`. +> your single best manipulation sequence toward top-level +> `terminated == true`. > You MAY recover *within* this one episode (re-pre-position, re-`pi0_pick` a > missed grasp, walk the Pi0 prompt ladder, `rotate_pitch`/`move_pose`) — that is > all one continuous attempt — but the instant you would want to reset/start over, > **STOP instead and write the audit** (success or honest -> `libero_terminated:false`). Do NOT call `reset`. Use the PROVEN LEVERS below to +> `terminated:false`). Do NOT call `reset`. Use the PROVEN LEVERS below to > get the single attempt right the first time.""" PROVEN_LEVERS = """These are battle-tested on seed 0 of THIS suite. You are now running a DIFFERENT @@ -54,7 +55,7 @@ a HIGH `lift_thresh` (e.g. 999) + `gripper_closed_thresh:0` turns it into a generic closed-loop CONTACT skill (used to turn the stove knob). - **`pi0_doubled`** = Pi0 closed-loop CONTACT skill (success := - `libero_terminated`). Use it for drawer/door open-close AND insertions; call it + `terminated`). Use it for drawer/door open-close AND insertions; call it repeatedly. DISAMBIGUATION / TARGETING: @@ -70,7 +71,7 @@ weak at reading side labels or distinguishing similar grocery items (ketchup/BBQ/tomato sauce, soup cans, cream cheese/butter). Do NOT let the wrist freely re-identify a non-basket target; it often locks onto a look-alike. - Instead: choose the target from `image_cam_hi_NN.png`, compute its agentview + Instead: choose the target from `agentview_high.png`, compute its agentview xyz, move over that candidate, project/track that SAME candidate in wrist, and refine only its surface/center coordinates. SAM3 scores ~0.02-0.06 on brand nouns ("alphabet soup", "tomato sauce") — prompt by colour+shape ("the short @@ -130,7 +131,7 @@ - mug→microwave + close (t9): the only UNSOLVED seed-0 cell — the round mug-in-hand walls ~3cm short of the In() threshold (deep narrow cavity). Try every lever (`pi0_doubled`, `move_pose`, push) and if it still walls, write an honest - `libero_terminated:false` with the max eef-y reached.""" + `terminated:false` with the max eef-y reached.""" RUNTIME = """A server process (`env_server.py`) is already running. It has Pi0.5 loaded and a single-env LIBERO sim. The runner manages the server and exposes structured @@ -141,63 +142,38 @@ - Call the real structured tools exposed by the runtime. - Use bare tool names in this prompt: `move_to`, `pi0_pick`, `release`, `set_gripper`, `rotate_wrist`, `rotate_pitch`, `move_pose`, `pi0_doubled`, - `view_driver_state`, `view_camera_meta`, `back_project`, `segment`, + `view_env_state`, `view_camera_meta`, `back_project`, `segment`, `read_text_file`, `write_text_file`, `list_dir`, `finish`. - Under some runtimes these same tools may appear namespaced; call the actual tool name shown in your tool list, preserving the same arguments and semantics. -The toolkit writes artifacts in `{{output_dir}}/`: - -- `{{output_dir}}/states.json` — top-level JSON array; each entry has - `step_idx`, `task_language`, `libero_terminated`, `state` (robot - proprioception + object_names; NO object coordinates), `command`, `result`, - `elapsed_s`, and world-map path fields when available. -- `{{output_dir}}/images/image_NN.png` — agentview RGB, 180°-rotated (Pi0 frame; - do NOT use for back-projection). -- `{{output_dir}}/images_cam/image_cam_NN.png` — agentview RGB in the CALIBRATION - frame; use for low-resolution pixel checks. -- `{{output_dir}}/depths/depth_NN.npy` — agentview metric depth (meters), - calibration frame. -- `{{output_dir}}/world/world_NN.npy` — HxWx3 precomputed world xyz per 256px - agentview pixel. Prefer `back_project`; read this manually only for debugging - or if the tool is unavailable. -- `{{output_dir}}/images_wrist/image_wrist_NN.png` — wrist RGB, calibration frame. -- `{{output_dir}}/depths_wrist/depth_wrist_NN.npy` — wrist metric depth (meters). -- `{{output_dir}}/world_wrist/world_wrist_NN.npy` — wrist world xyz map in the - SAME world frame as agentview. -- `{{output_dir}}/wrist_meta/wrist_meta_NN.json` — wrist intrinsics + extrinsic - FOR THAT STEP ONLY (the wrist cam moves, so it changes every step). -- `{{output_dir}}/images_cam_hi/image_cam_hi_NN.png` — HI-RES (1024x1024) - agentview RGB in calibration frame. USE THIS to inspect the scene and identify - objects — a far object spans 4x more pixels than at 256. -- `{{output_dir}}/world_hi/world_hi_NN.npy` — 1024x1024x3 float16 precomputed - world xyz per hi-res agentview pixel. Prefer `back_project`; if you manually - inspect it, never index a low-res pixel into this grid or vice versa. -- `{{output_dir}}/images_wrist_hi/image_wrist_hi_NN.png` / - `{{output_dir}}/world_wrist_hi/world_wrist_hi_NN.npy` — same hi-res pair for - the WRIST cam. - ⚠ Hi-res pixel (row,col) indexes ONLY the hi-res world map (and 256 pixel -> - 256 map). Don't mix grids; if you must convert, divide hi coords by 4. - ⚠ Hi-res files keep only the LAST 5 STEPS (disk); for older before/after - comparisons use the 256 files or `states.json` history. -- `{{output_dir}}/camera_meta.json` — agentview intrinsics K, cam->world - extrinsic, projection recipe. -- `{{output_dir}}/action_videos/step_NN_.mp4` — per-action clips generated - when the Dashboard is enabled. - -NN is zero-padded sequential (`00`, `01`, `02`, ...). Initial state step `00` is -dumped before you begin. Use `view_driver_state({"step": 0})` to read it.""" - -GOAL = """YOUR GOAL: produce `state.libero_terminated == true` in ONE episode. ⛔ NO +Each state record exposes `step`, top-level `task_language`, +`terminated`, `truncated`, coord-free `state`, an `artifacts` +list of logical base names, and a `log`. Storage paths are internal; do not +construct or parse them. + +Canonical observation keys include `agentview_policy.png`, `agentview.png`, +`agentview_high.png`, `agentview_depth.npy`, `agentview_world.npy`, +`agentview_world_high.npy`, `agentview_metadata.json`, `wrist.png`, +`wrist_high.png`, `wrist_depth.npy`, `wrist_world.npy`, +`wrist_world_high.npy`, and `wrist_metadata.json`. + +Use `view_env_state` to retrieve a record. It embeds the policy image and the +best available agentview and wrist images, preferring the matching `*_high.png` +artifact. Use `view_camera_meta`, `back_project`, and `segment` to consume +metadata and world maps without opening artifacts directly. Step `0` is the +initial state; step `-1` selects the latest state.""" + +GOAL = """YOUR GOAL: produce top-level `terminated == true` in ONE episode. ⛔ NO `reset`, NO retry (SINGLE-ATTEMPT MODE — see the override at the very top; it supersedes any reset/retry wording in the Rules below).""" RULES = """Rule 0 — USE IMAGES. After every primitive tool call, inspect the returned state - and image paths. If you need a state again, call `view_driver_state`. Read the - new `image_cam_hi_NN.png` path (calibration frame — the one you pick pixels in) - and, when close to a target, the `image_wrist_hi_NN.png` path. The image is - your spatial-reasoning input; `states.json` only gives proprioception + object - names. + and embedded images. If you need a state again, call `view_env_state`. Inspect + `agentview_high.png` (the calibration-frame image used for pixel selection) + and, when close to a target, `wrist_high.png`. The image is + your spatial-reasoning input; the returned `state` field only gives + proprioception + object names. Rule 1 — Pi0 is ONLY for the grasp. Use: pi0_pick({ @@ -215,19 +191,19 @@ Rule 1b — JUDGE THE GRASP from perception, NOT from a name. After a pick, decide "did I grab the target?" from two coord-free signals: • GRIPPER (proprioception): `state.robot0_gripper_qpos` from the latest - `states.json` entry — fingers closed but NOT fully shut (~0.01–0.05 gap) + state record — fingers closed but NOT fully shut (~0.01–0.05 gap) ⇒ holding an object; fully closed (~0.0) ⇒ grasped air. - • WRIST CAM: Read `image_wrist_hi_NN.png` after lifting. The target should + • WRIST CAM: inspect the embedded `wrist_high.png` image after lifting. The target should now be raised into the gripper, and the spot it came from should be EMPTY. Compare before/after wrist or agentview evidence; if needed, use `back_project` on wrist pixels to confirm the target surface z jumped up. `pi0_pick.success` (eef-lift + gripper-closure heuristic) is a HINT, not proof — always confirm with the wrist cam before carrying. -Rule 2 — Inspect THEN act. Call `view_driver_state({"step": 0})`, read the - returned high-resolution image path(s), and inspect the relevant memory/guides - BEFORE your first primitive. **Your task is `states.json[0]["task_language"]` - — read it and obey it verbatim.** This is the authoritative instruction (the BDDL +Rule 2 — Inspect THEN act. Call `view_env_state({"step": 0})`, inspect + `agentview_high.png` and the relevant memory/guides BEFORE your first + primitive. **Your task is the returned `task_language`; read it and obey it + verbatim.** This is the authoritative instruction (the BDDL `:language` tag). Do NOT infer the task from object names, from sibling recipes, or by guessing a task_map index — those caused wrong-task runs in the past. @@ -257,7 +233,7 @@ plate, a stove burner/cook-region, a wooden-cabinet top, and a pot lid all read as "flat disc at table height" in back-projected coordinates. They are only separable in the RGB. So before you carry-and-release onto a surface, look at - `image_cam_hi_NN.png` (and the wrist `image_wrist_hi_NN.png` once close) and + `agentview_high.png` (and `wrist_high.png` once close) and NAME each candidate surface: • PLATE ⇒ ceramic disc, usually white, with a clean raised rim (often colored concentric rings). This is the place target for "place it on the plate". @@ -284,7 +260,7 @@ `rotate_pitch`/`move_pose`) — that is still one continuous attempt — but you may NOT restart the episode. When the task terminates, OR when your single best sequence is exhausted (you'd otherwise want to reset), STOP and write the audit - (success or honest `libero_terminated:false`), then call `finish`. + (success or honest `terminated:false`), then call `finish`. NO teleport primitives (set_object_pose / articulate_to / js_move_to / carry_object — deleted/forbidden; a goal past OSC reach is approached physically or honestly reported, never warped). NO object world coords are @@ -292,8 +268,8 @@ LOCALIZATION = """This is the core of perception-isolated mode. To find where an object is: -1. Look at `image_cam_hi_NN.png` (1024x1024 — PREFER THIS; fall back to the - 256 `image_cam_NN.png` only if the hi file is absent) and find the target +1. Look at `agentview_high.png` (1024x1024 — PREFER THIS; fall back to + `agentview.png` only if the high-resolution artifact is absent) and find the target object's pixel (row, col). (row = vertical/y from top, col = horizontal/x from left.) 2. Call `back_project` on that pixel: @@ -316,7 +292,8 @@ metres away. Pick pixels firmly on the object's top surface.) ALWAYS apply the manipulation offsets from memory to the PERCEIVED position -(e.g. BOWL: eef_y = plate_y + 0.045). Verify visually in image_cam after moving.""" +(e.g. BOWL: eef_y = plate_y + 0.045). Verify visually in `agentview_high.png` +after moving.""" PERCEPTION_ALGORITHM = """This is the default perception algorithm for EVERY cell (from the 80-task localization sweep: `agentview_identity_wrist_geometry_except_basket`). @@ -335,11 +312,11 @@ ALGORITHM (run this BEFORE manipulating): -1. From `states.json[0]["task_language"]` + `image_cam_hi_00.png` + +1. From the initial `task_language` + `agentview_high.png` + object_names, infer the task-relevant TARGETS and DESTINATIONS (language only; never BDDL/poses). -2. GLOBAL SEMANTIC PASS (agentview hi-res): in `image_cam_hi_NN.png` choose each +2. GLOBAL SEMANTIC PASS (agentview hi-res): in `agentview_high.png` choose each target/destination candidate by RGB, label/shape, and global spatial relation. For duplicates (two bowls/plates/mugs) pick by RELATION (on stove, on cookie box, left/right/front/back), not `_1/_2`. For sauce/can/box groceries use the @@ -348,7 +325,7 @@ burner vs cabinet/drawer vs basket) semantically in RGB here. 3. COARSE XYZ (agentview): pick 3-8 pixels firmly on the chosen candidate in - `image_cam_hi_NN.png`, call `back_project` on the SAME pixels, take the median. + `agentview_high.png`, call `back_project` on the SAME pixels, take the median. Avoid edges/holes/shadows/table-gaps. This median is the IDENTITY ANCHOR for that entity. @@ -430,9 +407,9 @@ with older/oracle assumptions; do NOT copy coordinates and do NOT replay stale command lists. Re-derive every coordinate from THIS scene. """, - """INSPECT INITIAL STATE: call `view_driver_state({"step": 0})`; inspect -`task_language`, object_names, eef pose, `image_cam_hi_00.png`, -`image_wrist_hi_00.png` if useful, and `camera_meta.json`. Identify ALL target + """INSPECT INITIAL STATE: call `view_env_state({"step": 0})`; inspect + `task_language`, object_names, eef pose, `agentview_high.png`, + `wrist_high.png` if useful, and call `view_camera_meta` if needed. Identify ALL target objects, destination surfaces, and relation landmarks named by task_language. """, """RUN THE MANDATORY PRE-TASK PERCEPTION PASS (FIRST-STEP ALGORITHM above) — @@ -450,10 +427,9 @@ pi0_pick({"prompt": "...", "max_chunks": 20, ...}) release({}) -Each primitive tool blocks until the next `states.json` entry is dumped and -returns the new state view + log + image paths. Then inspect the returned state -+ high-resolution image paths (+ `back_project` as needed), decide, repeat -with NN=02, 03, ... +Each primitive tool blocks until the next state record is dumped and returns +the new state view, log, and embedded images. Inspect `agentview_high.png` and +`wrist_high.png` as needed, call `back_project` for geometry, decide, and repeat. """, """ALLOWED PRIMITIVES (physics-only; full schemas in the tool list/guides): `move_to`, `pi0_pick`, `pi0_doubled`, `release`, `set_gripper`, @@ -463,7 +439,7 @@ ⚠ INFRA NOTE: `pi0_doubled` IS implemented and callable in this runtime — verified. It runs the Pi0 VLA on a CONTACT skill (drawer/door open-close, knob -turn) with success := `libero_terminated` (no lift / no gripper-close +turn) with success := `terminated` (no lift / no gripper-close assumption — unlike `pi0_pick`). If ANY prior note or reference for this cell concluded that `pi0_doubled` is "unknown action" / missing / that drawer-or-door articulation is an unsolvable "structural dead-end" BECAUSE no @@ -475,16 +451,16 @@ SAM3 localization aid — `segment` (no robot motion): instead of eyeballing a pixel, call `segment({"prompt":"the black bowl on the cookies box", "camera":"agentview"})`. It runs SAM3 on the current image, back-projects the -mask via the matching world map, and writes `segments/segment_NN_XX.json` with a robust -median `world_xyz` (+ a `segments/segment_overlay_NN_XX.png` to confirm the right -object). Use `camera":"wrist"` (after parking the eef ~15–20 cm over the +mask via the matching world map, and returns a robust median `world_xyz` plus +logical `segment_artifact` and `overlay_artifact` names. Inspect the embedded +overlay image to confirm the right object. Use `camera":"wrist"` (after parking the eef ~15–20 cm over the target) for ±1–2 cm refinement, or `"point":[row,col]` for a point prompt. Text `prompt` and `point` are mutually exclusive; provide exactly one. ⚠ PROMPT PHRASING (SAM3 is sensitive): use a plain colour+shape+RELATION phrase, NEVER the internal/brand name from `object_names`/BDDL. `"the akita black bowl"` scores ~0.03 (SAM3 can't ground "akita") whereas `"the black bowl on the stove"` scores ~0.76. Strip proper nouns (akita, glazed_rim_porcelain_…) — say what it -LOOKS LIKE + where it is. Always inspect the returned overlay path to confirm +LOOKS LIKE + where it is. Always inspect the returned overlay image to confirm the mask landed on the right object before moving. This is a CONVENIENCE alternative to manual back-projection — if it returns `{"error":..., "fallback":...}` (server down / low score / no detection), walk @@ -497,16 +473,16 @@ traversals into <0.30 xy waypoints; for a door/drawer/knob use a SHORT capped OSC push or `pi0_doubled`, never one long push — it NaNs MuJoCo. If the task is unrecoverable within this one episode, do NOT reset — write an honest -stuck-audit (`libero_terminated:false`) and call `finish`. Never warp. +stuck-audit (`terminated:false`) and call `finish`. Never warp. """, - """WHEN state.libero_terminated == True: + """WHEN top-level `terminated == true` in the latest tool result: a. Write audit `{{output_dir}}/{{recipe_tag}}.json` with: suite, task_id, seed, regime:"strict_perception", strategy_notes (incl. how you localized), pick_result, final_state (latest state's `state`), - libero_terminated:true. + terminated:true. b. Call `finish`. If your single attempt does not solve it, write `{{output_dir}}/{{recipe_tag}}.json` with -libero_terminated:false + strategy_notes describing what you tried in this one +terminated:false + strategy_notes describing what you tried in this one episode and where it stalled. Then call `finish`. (NO reset, NO second attempt.)""", ) @@ -519,7 +495,7 @@ OUTPUT_DISCIPLINE = """- Brief reasoning before each tool call (1-2 sentences): observation → decision. - Don't re-read files already in this session. -- Don't call `view_driver_state` immediately after a primitive tool already +- Don't call `view_env_state` immediately after a primitive tool already returned the new state. - Save the audit BEFORE calling `finish`. - Stop immediately after writing the audit and calling `finish`. Do not chat further.""" diff --git a/robots/libero/prompts/user.py b/robots/libero/prompts/user.py index 668a4db4..a4437e9d 100644 --- a/robots/libero/prompts/user.py +++ b/robots/libero/prompts/user.py @@ -10,9 +10,10 @@ - recipe: {{output_dir}}/recipe_{{recipe_tag}}.jsonl""" -MODE = """Use the high-resolution image paths returned by view_driver_state and -back_project to localize objects before motion.""" +MODE = """Inspect `agentview_high.png` returned by `view_env_state`, then use +`back_project` or `segment` to localize objects before motion.""" -BEGIN = """read MEMORY.md, the guides, then `view_driver_state({"step":0})` and the -returned high-resolution images. Localize the target, then plan and execute.""" +BEGIN = """Read MEMORY.md and the guides, then call +`view_env_state({"step": 0})` and inspect `agentview_high.png`. Localize the +target, then plan and execute.""" diff --git a/robots/libero/toolkit.py b/robots/libero/toolkit.py index 475c1e74..a9e22dc2 100644 --- a/robots/libero/toolkit.py +++ b/robots/libero/toolkit.py @@ -5,34 +5,32 @@ """ from __future__ import annotations -import shutil -import time from functools import partial from typing import Any from robots.libero import tools as libero_tools -from rpent.dashboard.events import DashboardEventSink, ToolResultEvent -from rpent.tools.toolkit import ToolCancelled, Toolkit +from rpent.dashboard.events import DashboardEventSink +from rpent.tools.state import EnvState +from rpent.tools.toolkit import Toolkit from rpent.utils.logging import get_logger, get_output_dir class LiberoToolkit(Toolkit): """Toolkit for the LIBERO environment.""" - # Tool schemas keyed by name (built once from the canonical ordered list - # in libero_tools.TOOLS_SPEC) so each tool registers with its own spec. - _SPECS = {spec["name"]: spec for spec in libero_tools.TOOLS_SPEC} + _FRAME_ARTIFACTS = { + "camera": "agentview.png", + "wrist": "wrist.png", + } def __init__( self, *, primitives_kwargs: dict[str, Any], dashboard_events: DashboardEventSink, - video_path: str | None = None, ) -> None: - super().__init__(dashboard_events=dashboard_events) - self._next_step: int = 0 - self._video_path: str | None = video_path + state = EnvState(get_output_dir()) + super().__init__(dashboard_events=dashboard_events, state=state) self.init_primitives_clean(primitives_kwargs=primitives_kwargs) self._register_libero_tools() @@ -40,67 +38,64 @@ def __init__( # Registration # ------------------------------------------------------------------ def _register_libero_tools(self) -> None: - specs = self._SPECS - # Inspection tools do not advance environment state. Most are stateless - # module functions; segment is bound to the primitives-owned SAM3 client. - inspection_handlers = { - "view_driver_state": libero_tools.view_driver_state, - "view_camera_meta": libero_tools.view_camera_meta, - "back_project": libero_tools.back_project, - "segment": self._primitives.segment, + # These read-only handlers need the run's EnvState bound in. Every + # other spec binds to a primitive-driver method and captures state by + # default unless that method is explicitly marked @readonly. + state_handlers = { + "view_env_state": partial( + libero_tools.view_env_state, state=self._state + ), + "view_camera_meta": partial( + libero_tools.view_camera_meta, state=self._state + ), + "back_project": partial(libero_tools.back_project, state=self._state), + "segment": partial(self._primitives.segment, state=self._state), } - for name, handler in inspection_handlers.items(): - self.add_tool(name, specs[name], handler) - # Primitive tools: each goes through _step, which looks up the - # matching primitive method via getattr at call time. - for name in libero_tools.PRIMITIVE_TOOL_NAMES: - self.add_tool(name, specs[name], partial(self._step, name)) - - def _step(self, name: str, **kwargs) -> dict: - """Run ``self._primitives.(**kwargs)``, dump the new step, and - return the rendered state view + log. - """ - command = {"action": name, **kwargs} - t0 = time.time() - start_frame = self._primitives.recorded_frame_count() - try: - result = getattr(self._primitives, name)(**kwargs) - self.raise_if_cancelled() - except ToolCancelled as exc: - result = { - "error": str(exc), - "code": "tool_cancelled", - "interrupted": True, - } - elapsed = round(time.time() - t0, 2) - - if isinstance(result, dict): - result_dict = result - else: - result_dict = {"value": result} - - self._next_step += 1 - step_idx = self._next_step - output_dir = get_output_dir() + for spec in libero_tools.TOOLS_SPEC: + name = spec["name"] + if name in state_handlers: + handler = state_handlers[name] + else: + handler = getattr(self._primitives, name, None) + if handler is None: + continue # spec without a backing primitive method + self.add_tool(name, spec, handler) + + def get_env_state( + self, + *, + command: dict[str, Any], + result: dict[str, Any], + elapsed_s: float, + ) -> dict[str, Any]: + frame_start = self._action_frame_cursor + self._action_frame_cursor = self._primitives.recorded_frame_count() + record = libero_tools.dump_state( + self._primitives, + self._state, + log={"command": command, "result": result, "elapsed_s": elapsed_s}, + ) if self._dashboard_events.enabled: - video_dir = libero_tools.artifact_path(output_dir, "action_videos") - video_path = video_dir / f"step_{step_idx:02d}_{name}.mp4" try: - self._primitives.save_frame_slice(start_frame, str(video_path), fps=20) + frames = self._primitives.frame_slice(frame_start) + if frames: + candidate = f"action_{command['action']}.mp4" + self._state.save( + candidate, + frames, + step=record.step_idx, + fps=20, + ) except Exception as e: get_logger("libero_toolkit").warning( - f"failed to save action clip to {video_path}: {e}" + "failed to save action clip for step %s: %s", + record.step_idx, + e, ) - libero_tools.dump_state( - self._primitives, - str(output_dir), - step_idx=step_idx, - log={"command": command, "result": result_dict, "elapsed_s": elapsed}, - ) - out = libero_tools.view_driver_state(step_idx) - out["agent_elapsed_s"] = elapsed - if result_dict.get("interrupted"): - out.update(result_dict) + out = libero_tools.view_env_state(record.step_idx, state=self._state) + out["agent_elapsed_s"] = elapsed_s + if result.get("interrupted"): + out.update(result) return out def init_primitives_clean( @@ -109,19 +104,7 @@ def init_primitives_clean( primitives_kwargs: dict[str, Any], ) -> None: """Wipe stale run artifacts, build the LiberoPrimitives, dump step 0.""" - out_dir = get_output_dir() - out_dir.mkdir(parents=True, exist_ok=True) - for sub in libero_tools.ARTIFACT_DIRECTORIES: - target = out_dir / sub - if target.exists(): - shutil.rmtree(target) - for target in ( - libero_tools.artifact_path(out_dir, "states"), - libero_tools.artifact_path(out_dir, "metadata", camera="agentview", resolution="low"), - libero_tools.artifact_path(out_dir, "episode_video"), - ): - if target.exists(): - target.unlink() + self._state.reset() primitives = libero_tools.LiberoPrimitives( check_cancelled=self.raise_if_cancelled, @@ -129,30 +112,22 @@ def init_primitives_clean( ) primitives.reset() primitives.start_recording() - libero_tools.dump_state(primitives, str(out_dir), step_idx=0, log=None) - self._dashboard_events.emit( - ToolResultEvent( - name="view_driver_state", - result=libero_tools.view_driver_state(0), - ) - ) - + self._action_frame_cursor = primitives.recorded_frame_count() + record = libero_tools.dump_state(primitives, self._state, log=None) self._primitives = primitives + self._publish_step(record) def close(self) -> None: - """Flush the agent-side video buffer to disk (end-of-run). - """ - if self._video_path is None: - return + """Flush the agent-side video buffer through ``EnvState``.""" try: - self._primitives.stop_recording_and_save(self._video_path) + frames = self._primitives.stop_recording() + if frames: + self._state.save("episode.mp4", frames, step=None, fps=20) except Exception as e: - # The runner is in the cleanup path; never let a video save - # abort it. get_logger("libero_toolkit").warning( - f"failed to save video to {self._video_path}: {e}" + f"failed to save episode video: {e}" ) def write_recipe(self, recipe_tag: str) -> str: """Write the LIBERO recipe JSONL from the dumped state trace.""" - return libero_tools.write_recipe_from_states(str(get_output_dir()), recipe_tag) + return libero_tools.write_recipe_from_states(self._state, recipe_tag) diff --git a/robots/libero/tools.py b/robots/libero/tools.py index 44145227..921b9ab9 100644 --- a/robots/libero/tools.py +++ b/robots/libero/tools.py @@ -1,86 +1,20 @@ """LIBERO + OpenPI tool implementation.""" from __future__ import annotations -import json -import os from collections.abc import Callable -from pathlib import Path from typing import Any -import imageio.v2 as imageio import numpy as np from robots.libero.env_client import LiberoEnvClient -from rpent.utils.logging import get_logger, get_output_dir +from rpent.tools.state import EnvState, StepRecord +from rpent.tools.toolkit import readonly +from rpent.utils.logging import get_logger from rpent.utils.sam3_client import Sam3Client from rpent.utils.vla_client import VLAClient logger = get_logger("libero") -ARTIFACT_LAYOUT: dict[tuple[str | None, str | None, str], str] = { - ("agentview", "low", "policy_image"): "images/image_{step:02d}.png", - ("agentview", "low", "image"): "images_cam/image_cam_{step:02d}.png", - ("agentview", "low", "depth"): "depths/depth_{step:02d}.npy", - ("agentview", "low", "world"): "world/world_{step:02d}.npy", - ("agentview", "low", "metadata"): "camera_meta.json", - ("agentview", "high", "image"): "images_cam_hi/image_cam_hi_{step:02d}.png", - ("agentview", "high", "world"): "world_hi/world_hi_{step:02d}.npy", - ("wrist", "low", "image"): "images_wrist/image_wrist_{step:02d}.png", - ("wrist", "low", "depth"): "depths_wrist/depth_wrist_{step:02d}.npy", - ("wrist", "low", "world"): "world_wrist/world_wrist_{step:02d}.npy", - ("wrist", "low", "metadata"): "wrist_meta/wrist_meta_{step:02d}.json", - ("wrist", "high", "image"): "images_wrist_hi/image_wrist_hi_{step:02d}.png", - ("wrist", "high", "world"): "world_wrist_hi/world_wrist_hi_{step:02d}.npy", - (None, None, "states"): "states.json", - (None, None, "episode_video"): "episode.mp4", - (None, None, "segments"): "segments", - (None, None, "action_videos"): "action_videos", -} -ARTIFACT_DIRECTORIES: tuple[str, ...] = ( - "images", - "images_cam", - "depths", - "world", - "images_cam_hi", - "world_hi", - "images_wrist", - "depths_wrist", - "world_wrist", - "wrist_meta", - "images_wrist_hi", - "world_wrist_hi", - "segments", - "action_videos", -) - - -def artifact_path( - output_dir: str | os.PathLike[str], - kind: str, - *, - step: int | None = None, - camera: str | None = None, - resolution: str | None = None, -) -> Path: - """Resolve one artifact path from the shared layout. - - ``kind`` identifies the artifact; the remaining fields are optional - qualifiers and must be passed by keyword to avoid mixing them up. - """ - return Path(output_dir) / _artifact_relative_path(step, camera, resolution, kind) - - -def _artifact_relative_path( - step: int | None, - camera: str | None, - resolution: str | None, - kind: str, -) -> str: - template = ARTIFACT_LAYOUT[(camera, resolution, kind)] - if step is None and "{step" in template: - raise ValueError(f"{kind} artifact requires a step") - return template.format(step=step) - def _normalize_xyz(xyz): """Coerce an LLM-supplied xyz into a length-3 list[float].""" @@ -116,7 +50,7 @@ def __init__( self._last_obs_eef_z = None self._last_obs_gripper = None # Per-env-step frame buffer for diagnostic video rendering. - # Toggled via start_recording() / stop_recording_and_save(). + # Toggled via start_recording() / stop_recording(). self._recording = False self._frames = [] @@ -131,22 +65,14 @@ def record_frame(self, obs): def recorded_frame_count(self) -> int: return len(self._frames) - def stop_recording_and_save(self, path: str, fps: int = 20): - os.makedirs(os.path.dirname(path) or ".", exist_ok=True) - n = len(self._frames) - if n > 0: - imageio.mimwrite(path, self._frames, fps=fps) + def stop_recording(self) -> list[np.ndarray]: + frames = list(self._frames) self._recording = False self._frames = [] - return {"path": path, "n_frames": n} + return frames - def save_frame_slice(self, start: int, path: str, fps: int = 20): - os.makedirs(os.path.dirname(path) or ".", exist_ok=True) - frames = list(self._frames[int(start):]) - n = len(frames) - if n > 0: - imageio.mimwrite(path, frames, fps=fps) - return {"path": path, "n_frames": n, "fps": fps} + def frame_slice(self, start: int) -> list[np.ndarray]: + return list(self._frames[int(start):]) def set_obs(self, obs): self._last_obs = obs @@ -247,8 +173,8 @@ def pi0_pick( if descent_done and ascended and closed: success = True break - if self.env.episode_terminated or self.env.episode_truncated: - success = self.env.episode_terminated + if self.env.terminated or self.env.truncated: + success = self.env.terminated break return { @@ -260,7 +186,8 @@ def pi0_pick( "peak_lift_m": post_min_peak_z - min_z, # actual post-descent ascent "min_gripper_opening": min_grip, "final_gripper_opening": last_grip, - "libero_terminated": self.env.episode_terminated, + "terminated": self.env.terminated, + "truncated": self.env.truncated, "diagnostics": { "start_eef_z": round(start_z, 4), "peak_eef_z": round(peak_z, 4), @@ -293,8 +220,8 @@ def pi0_doubled( for c in range(max_chunks): self._vlm_chunk(instr) chunks_used = c + 1 - if self.env.episode_terminated or self.env.episode_truncated: - task_success = self.env.episode_terminated + if self.env.terminated or self.env.truncated: + task_success = self.env.terminated break return { @@ -305,9 +232,10 @@ def pi0_doubled( "contact_skill_executed": chunks_used > 0, "chunks_used": chunks_used, "max_chunks": max_chunks, - "libero_terminated": self.env.episode_terminated, + "terminated": self.env.terminated, + "truncated": self.env.truncated, "diagnostics": { - "mode": "contact_skill_success_by_libero_terminated", + "mode": "contact_skill_success_by_termination", "success_meaning": ( "`success` mirrors official LIBERO task termination only; " "for intermediate contact skills, inspect image/state evidence." @@ -366,7 +294,7 @@ def move_to( action[5] = float(np.clip(step_dyaw / 0.10, -1.0, 1.0)) action[6] = gripper self._step_env(action) - if self.env.episode_terminated or self.env.episode_truncated: + if self.env.terminated or self.env.truncated: break final = self._last_obs_eef_pos return { @@ -376,7 +304,8 @@ def move_to( "final_dist_m": round(float(np.linalg.norm(target - final)), 4), "steps_used": len(traj), "max_steps": max_steps, - "libero_terminated": self.env.episode_terminated, + "terminated": self.env.terminated, + "truncated": self.env.truncated, } def rotate_wrist( @@ -439,7 +368,7 @@ def _yaw_of(quat_xyzw): action[5] = float(np.clip(action[5], -1.0, 1.0)) action[6] = float(gripper) self._step_env(action) - if self.env.episode_terminated or self.env.episode_truncated: + if self.env.terminated or self.env.truncated: break final_yaw = _yaw_of(self.env.raw_obs()["robot0_eef_quat"]) return { @@ -449,7 +378,8 @@ def _yaw_of(quat_xyzw): "final_yaw": round(final_yaw, 4), "final_err": round(float((target_yaw - final_yaw + np.pi) % (2 * np.pi) - np.pi), 4), "steps_used": len(traj), - "libero_terminated": self.env.episode_terminated, + "terminated": self.env.terminated, + "truncated": self.env.truncated, } def rotate_pitch( @@ -519,7 +449,7 @@ def _pitch_of(quat_xyzw): action[3] = float(np.clip(action[3], -1.0, 1.0)) action[6] = float(gripper) self._step_env(action) - if self.env.episode_terminated or self.env.episode_truncated: + if self.env.terminated or self.env.truncated: break final_pitch = _pitch_of(self.env.raw_obs()["robot0_eef_quat"]) return { @@ -530,7 +460,8 @@ def _pitch_of(quat_xyzw): "final_err": round(float( (target_pitch - final_pitch + np.pi) % (2 * np.pi) - np.pi), 4), "steps_used": len(traj), - "libero_terminated": self.env.episode_terminated, + "terminated": self.env.terminated, + "truncated": self.env.truncated, } def move_pose( @@ -590,7 +521,7 @@ def _yaw_of(q): action[5] = float(np.clip(np.clip(y_err, -yaw_step, yaw_step) / 0.10, -1.0, 1.0)) action[6] = float(gripper) self._step_env(action) - if self.env.episode_terminated or self.env.episode_truncated: + if self.env.terminated or self.env.truncated: break final = self._last_obs_eef_pos fq = self.env.raw_obs()["robot0_eef_quat"] @@ -600,7 +531,8 @@ def _yaw_of(q): "final_dist_m": round(float(np.linalg.norm(target - final)), 4), "final_pitch": round(_pitch_of(fq), 4), "steps_used": step + 1, - "libero_terminated": self.env.episode_terminated, + "terminated": self.env.terminated, + "truncated": self.env.truncated, } def release( @@ -620,7 +552,7 @@ def release( action[6] = -1.0 # open self._step_env(action) peak_grip = max(peak_grip, self._last_obs_gripper) - if self.env.episode_terminated or self.env.episode_truncated: + if self.env.terminated or self.env.truncated: break return { "name": "release", @@ -628,7 +560,8 @@ def release( "start_gripper_opening": round(start_grip, 4), "peak_gripper_opening": round(peak_grip, 4), "final_gripper_opening": round(self._last_obs_gripper, 4), - "libero_terminated": self.env.episode_terminated, + "terminated": self.env.terminated, + "truncated": self.env.truncated, } def set_gripper( @@ -644,24 +577,28 @@ def set_gripper( action = np.zeros(7, dtype=np.float32) action[6] = g self._step_env(action) - if self.env.episode_terminated or self.env.episode_truncated: + if self.env.terminated or self.env.truncated: break return { "name": "set_gripper", "gripper": g, "steps": n, - "libero_terminated": self.env.episode_terminated, + "terminated": self.env.terminated, + "truncated": self.env.truncated, } # ---- introspection helpers (for LLM-in-the-loop) ---- + @readonly def segment( self, prompt: str = "", camera: str = "agentview", - step: int | None = None, + step: int = -1, point: list[int] | None = None, min_score: float = 0.2, + *, + state: EnvState, ) -> dict: """Call SAM3 on an existing image artifact without advancing the env. @@ -669,9 +606,11 @@ def segment( artifacts. Errors are structured so the agent can continue with image inspection and ``back_project``. """ - nn = _latest_step() if step is None else int(step) - if nn is None: - return {"error": "no state entries; cannot select segment image"} + try: + record = state.get(step) + except Exception as exc: + return {"error": f"state step not available: {exc}"} + nn = record.step_idx camera = camera or "agentview" prompt = prompt.strip() @@ -680,26 +619,27 @@ def segment( if has_prompt == has_point: return {"error": "segment needs exactly one of prompt or point"} try: - image_path, world_path, artifact_pairs = _select_segment_artifacts( - nn, camera + image_name, world_name, artifact_pairs = _select_segment_artifacts( + state, record, camera ) except ValueError as e: return {"error": str(e)} - if image_path is None: + if image_name is None or world_name is None: return { "error": "complete segment artifacts not found", "step": nn, "camera": camera, - "checked_paths": [ - str(path) + "checked_artifacts": [ + name for image, world in artifact_pairs - for path in (image, world) + for name in (image, world) + if name ], } try: data = self._sam3_client.segment( - image_path, + state.load_bytes(image_name, step=nn), text_prompt=prompt if has_prompt else None, point=point, min_score=min_score, @@ -709,36 +649,41 @@ def segment( "error": str(e), "step": nn, "camera": camera, - "image_path": str(image_path), + "image_artifact": image_name, } except Exception as e: return { "error": f"segmentation service call failed: {e}", "step": nn, "camera": camera, - "image_path": str(image_path), + "image_artifact": image_name, "fallback": "Use manual visual localization and back_project.", } - out_dir = get_output_dir() - segment_path, overlay_candidate_path, segment_index = ( - _next_segment_artifact_paths(out_dir, nn) - ) - overlay_path = None + segment_index = _next_segment_index(record) + segment_name = f"segment_{segment_index:02d}.json" + overlay_name = f"segment_overlay_{segment_index:02d}.png" + saved_overlay = None mask = data.mask if data.found and isinstance(mask, np.ndarray): - if world_path is None or not world_path.exists(): + try: + world_map = state.load(world_name, step=nn) + except Exception as exc: world_result = { "world_xyz": None, - "world_error": "world map artifact not found for selected image", - "expected_world_path": str(world_path) if world_path else None, + "world_error": f"world map artifact not available: {exc}", + "expected_world_artifact": world_name, } else: - world_result = _mask_to_world(mask, np.load(world_path)) - world_result["world_path"] = str(world_path) - overlay_path = overlay_candidate_path - if not _write_segment_overlay(image_path, mask, overlay_path): - overlay_path = None + world_result = _mask_to_world(mask, world_map) + world_result["world_artifact"] = world_name + overlay = _make_segment_overlay(state.load(image_name, step=nn), mask) + if overlay is not None and state.save( + overlay_name, + overlay, + step=nn, + ): + saved_overlay = overlay_name else: world_result = { "world_xyz": None, @@ -751,7 +696,7 @@ def segment( "camera": camera, "source_step": nn, "segment_index": segment_index, - "image_path": str(image_path), + "image_artifact": image_name, "min_score": min_score, "score": round(float(data.score), 3) if data.score is not None else None, "box": data.box, @@ -764,103 +709,97 @@ def segment( if not data.found: segment_blob["error"] = data.reason or "SAM3 found no mask" segment_blob.update(world_result) - segment_path.write_text(json.dumps(segment_blob, indent=2, default=str)) + saved_segment = state.save( + segment_name, + segment_blob, + step=nn, + ) result = { "found": data.found, "step": nn, "camera": camera, - "image_path": str(image_path), - "segment_path": str(segment_path), + "image_artifact": image_name, "score": segment_blob["score"], "box": segment_blob["box"], "world_xyz": segment_blob["world_xyz"], "world_error": segment_blob.get("world_error"), } - if "error" in segment_blob: + if saved_segment is None: + result["error"] = f"failed to persist segment artifact {segment_name}" + result["code"] = "segment_artifact_save_failed" + result["attempted_segment_artifact"] = segment_name + if "error" in segment_blob: + result["segmentation_error"] = segment_blob["error"] + result["fallback"] = "Use manual visual localization and back_project." + else: + result["segment_artifact"] = saved_segment + if saved_segment is not None and "error" in segment_blob: result["error"] = segment_blob["error"] result["fallback"] = "Use manual visual localization and back_project." - if overlay_path is not None and overlay_path.exists(): - result["overlay_path"] = str(overlay_path) + if saved_overlay is not None: + result["overlay_artifact"] = saved_overlay + result["_image_bytes"] = state.load_bytes(saved_overlay, step=nn) return result -# --------------------------------------------------------------------------- -# State artifacts -# --------------------------------------------------------------------------- +def _is_primitive_action(name: object) -> bool: + """Whether ``name`` is a state-advancing LIBERO primitive. - -def _append_state(output_dir: str, blob: dict) -> None: - """Append *blob* to ``/states.json`` atomically.""" - path = artifact_path(output_dir, "states") - tmp_path = path.parent / f"{path.name}.tmp" - if path.exists(): - with open(path) as f: - states = json.load(f) - else: - states = [] - states.append(blob) - with open(tmp_path, "w") as f: - json.dump(states, f, indent=2) - os.replace(tmp_path, path) + A primitive is any non-read-only method on :class:`LiberoPrimitives`; + read-only tools and non-strings read as ``False``. + """ + if not isinstance(name, str): + return False + method = getattr(LiberoPrimitives, name, None) + return method is not None and not bool(getattr(method, "_readonly", False)) -def write_recipe_from_states(output_dir: str, recipe_tag: str) -> str: - """Find a command sequence that gets ``libero_terminated=True``. +def write_recipe_from_states(state: EnvState, recipe_tag: str) -> str: + """Find a command sequence that gets ``terminated=True``. - Export non-error LIBERO primitive commands from ``states.json`` and - successful segment calls from ``segments/segment_*.json``. + Export non-error LIBERO primitive commands and successful segment calls. """ - states_path = artifact_path(output_dir, "states") - if states_path.exists(): - with open(states_path) as f: - states = json.load(f) - else: - states = [] - command_events = [] - for step_idx, entry in enumerate(states): - if not entry: - continue - command = entry.get("command") - if command is None: - continue - if command.get("action") not in PRIMITIVE_TOOL_NAMES: - continue - result = entry.get("result") - if isinstance(result, dict) and result.get("error"): - continue - command_events.append(((step_idx, -1), command)) - - for artifact in artifact_path(output_dir, "segments").glob("segment_*.json"): - with artifact.open() as f: - segment = json.load(f) - if segment.get("error"): - continue - if segment["mode"] == "text": - command = { - "action": "segment", - "prompt": segment["prompt"], - "camera": segment["camera"], - } - else: - command = { - "action": "segment", - "point": segment["point"], - "camera": segment["camera"], - } - source_step = int(segment["source_step"]) - event_order = (source_step, int(segment["segment_index"])) - command_events.append((event_order, command)) + for record in state.records(): + command = record.command + result = record.result + if ( + command is not None + and _is_primitive_action(command.get("action")) + and not (isinstance(result, dict) and result.get("error")) + ): + command_events.append(((record.step_idx, -1), command)) + + for name in sorted(record.artifacts): + if not (name.startswith("segment_") and name.endswith(".json")): + continue + segment = state.load(name, step=record.step_idx) + if segment.get("error"): + continue + if segment["mode"] == "text": + segment_command = { + "action": "segment", + "prompt": segment["prompt"], + "camera": segment["camera"], + } + else: + segment_command = { + "action": "segment", + "point": segment["point"], + "camera": segment["camera"], + } + event_order = (record.step_idx, int(segment["segment_index"])) + command_events.append((event_order, segment_command)) - recipe_path = os.path.join(output_dir, f"recipe_{recipe_tag}.jsonl") - tmp_path = recipe_path + ".tmp" command_events.sort(key=lambda event: event[0]) - with open(tmp_path, "w") as f: - for _, command in command_events: - f.write(json.dumps(command, separators=(",", ":")) + "\n") - os.replace(tmp_path, recipe_path) - return recipe_path + recipe_name = f"recipe_{recipe_tag}.jsonl" + state.save( + recipe_name, + [command for _, command in command_events], + step=None, + ) + return recipe_name def _metric_depth(depth: Any, camera_meta: dict) -> np.ndarray: @@ -889,39 +828,13 @@ def _world_from_depth(depth_metric: np.ndarray, camera_meta: dict) -> np.ndarray return (camera_points @ extrinsic.T)[..., :3] -def dump_state(primitives: LiberoPrimitives, output_dir: str, step_idx: int, - log: dict | None = None) -> dict: - """Dump state snapshot, images, and depth for step *step_idx*. - - Writes: - - ``/images/image_NN.png`` (Pi0-frame agentview) - - ``/images_cam/image_cam_NN.png`` (calibration-frame agentview) - - ``/depths/depth_NN.npy`` (metric depth, meters) - - ``/world/world_NN.npy`` (agentview world xyz map) - - ``/images_wrist/image_wrist_NN.png`` - - ``/depths_wrist/depth_wrist_NN.npy`` - - ``/world_wrist/world_wrist_NN.npy`` - - ``/wrist_meta/wrist_meta_NN.json`` - - high-res ``images_cam_hi`` / ``world_hi`` artifacts - - high-res ``images_wrist_hi`` / ``world_wrist_hi`` artifacts - - ``/camera_meta.json`` (static, once) - - appends the step blob to ``/states.json`` - - If *log* is provided (the return value of :func:`execute`), its - ``command``, ``result``, and ``elapsed_s`` fields are merged into the - step blob so a single entry captures everything. - """ - for directory in ARTIFACT_DIRECTORIES: - (Path(output_dir) / directory).mkdir(parents=True, exist_ok=True) - - agent_world_map = None - wrist_world_map = None - agent_world_map_hi = None - wrist_world_map_hi = None - # Reuse one raw observation snapshot for state and per-step artifacts. +def dump_state( + primitives: LiberoPrimitives, + env_state: EnvState, + log: dict | None = None, +) -> StepRecord: + """Save one Libero observation through its owned state record.""" raw = primitives.env.raw_obs() - # Expose robot proprioception and object names, but never privileged - # object coordinates; the agent must localize through visual artifacts. state = { "robot0_eef_pos": [float(x) for x in raw["robot0_eef_pos"]], "robot0_eef_quat": [float(x) for x in raw["robot0_eef_quat"]], @@ -932,9 +845,32 @@ def dump_state(primitives: LiberoPrimitives, output_dir: str, step_idx: int, if k.endswith("_pos") and "robot0" not in k and "to_robot" not in k ), } - imageio.imwrite( - artifact_path(output_dir, "policy_image", step=step_idx, camera="agentview", resolution="low"), + log = log or {} + with env_state.record_step( + state=state, + terminated=primitives.env.terminated, + truncated=primitives.env.truncated, + command=log.get("command"), + result=log.get("result"), + elapsed_s=log.get("elapsed_s"), + extras={ + "task_language": primitives.env.get_task_language(), + }, + ) as step_idx: + _save_observation_artifacts(primitives, env_state, step_idx, raw) + return env_state.get(step_idx) + + +def _save_observation_artifacts( + primitives: LiberoPrimitives, + env_state: EnvState, + step_idx: int, + raw: dict[str, Any], +) -> None: + env_state.save( + "agentview_policy.png", primitives._last_obs["main_images"], + step=step_idx, ) # --- camera calibration (static for agentview): fetch metadata as needed --- @@ -943,8 +879,7 @@ def dump_state(primitives: LiberoPrimitives, output_dir: str, step_idx: int, height=256, width=256, ) or {} - camera_meta_path = artifact_path(output_dir, "metadata", camera="agentview", resolution="low") - if agentview_meta and not camera_meta_path.exists(): + if agentview_meta: cam_meta_out = dict(agentview_meta) cam_meta_out["projection"] = ( "Prefer the back_project(row, col, step=NN) MCP tool; it " @@ -953,24 +888,30 @@ def dump_state(primitives: LiberoPrimitives, output_dir: str, step_idx: int, "256x256 calibration-frame image." ) cam_meta_out["note"] = ( - "depth_NN.npy is in this camera frame (vertical-flipped raw " - "buffer). image_NN.png is rotated 180deg (Pi0 convention) and " - "is NOT in the same frame as depth/K." + "The agentview_depth.npy observation is aligned with agentview.png. " + "agentview_policy.png uses the Pi0 orientation and must not supply " + "pixels for back-projection." + ) + env_state.save( + "agentview_metadata.json", + cam_meta_out, + step=step_idx, ) - with open(camera_meta_path, "w") as f: - json.dump(cam_meta_out, f, indent=2) # --- per-step RGB in the depth/K frame (vertical-flip of the raw buffer) --- - # The agent picks object pixels HERE (same frame as depth_NN.npy + K), so - # pixel -> depth -> back-project is direct. (image_NN.png is the 180°-rotated - # Pi0-convention frame and must NOT be used for back-projection.) + # Agentview pixels align with the matching depth and calibration. The policy + # image uses Pi0 orientation and must not supply back-projection pixels. try: ci = raw.get("agentview_image") if ci is not None: ci = np.asarray(ci) if ci.dtype != np.uint8: ci = ci.astype(np.uint8) - imageio.imwrite(artifact_path(output_dir, "image", step=step_idx, camera="agentview", resolution="low"), ci[::-1]) + env_state.save( + "agentview.png", + ci[::-1], + step=step_idx, + ) except Exception as e: logger.warning("image_cam dump failed: %s", e) @@ -983,21 +924,19 @@ def dump_state(primitives: LiberoPrimitives, output_dir: str, step_idx: int, # depth map in this frame. VERIFIED 5/5: projecting each GT object # world pos via M lands on a pixel whose depth_flip[row,col] matches # the object's surface depth (plate Δ6mm, cookies Δ14mm). So - # pixel(row,col) in depth_NN.npy back-projects correctly with - # camera_meta.json (NOT the same frame as the 180°-rotated - # image_NN.png — see camera_meta note). + # Pixels in agentview.png align with agentview_depth.npy and the + # per-step agentview metadata saved with the record artifacts. d = _metric_depth(d, agentview_meta)[::-1] - np.save( - artifact_path(output_dir, "depth", step=step_idx, camera="agentview", resolution="low"), + env_state.save( + "agentview_depth.npy", d.astype(np.float32), + step=step_idx, ) world = _world_from_depth(d, agentview_meta).astype(np.float32) - np.save( - artifact_path(output_dir, "world", step=step_idx, camera="agentview", resolution="low"), + env_state.save( + "agentview_world.npy", world, - ) - agent_world_map = _artifact_relative_path( - step_idx, "agentview", "low", "world" + step=step_idx, ) except Exception as e: logger.warning("depth dump failed: %s", e) @@ -1011,7 +950,11 @@ def dump_state(primitives: LiberoPrimitives, output_dir: str, step_idx: int, wimg = np.asarray(wimg) if wimg.dtype != np.uint8: wimg = wimg.astype(np.uint8) - imageio.imwrite(artifact_path(output_dir, "image", step=step_idx, camera="wrist", resolution="low"), wimg[::-1]) + env_state.save( + "wrist.png", + wimg[::-1], + step=step_idx, + ) except Exception as e: logger.warning("wrist image dump failed: %s", e) @@ -1031,31 +974,30 @@ def dump_state(primitives: LiberoPrimitives, output_dir: str, step_idx: int, logger.warning("wrist camera meta missing; skipping wrist depth/world") else: wdpt_metric = _metric_depth(wdpt_arr, wmeta)[::-1] - np.save( - artifact_path(output_dir, "depth", step=step_idx, camera="wrist", resolution="low"), + env_state.save( + "wrist_depth.npy", wdpt_metric.astype(np.float32), + step=step_idx, ) world_w = _world_from_depth(wdpt_metric, wmeta).astype(np.float32) - np.save( - artifact_path(output_dir, "world", step=step_idx, camera="wrist", resolution="low"), + env_state.save( + "wrist_world.npy", world_w, - ) - wrist_world_map = _artifact_relative_path( - step_idx, "wrist", "low", "world" + step=step_idx, ) wmeta_out = dict(wmeta) wmeta_out["note"] = ( "MOVING camera: extrinsic_cam2world is for THIS step " - "only. world_wrist_NN.npy[row,col] gives world " - "(x,y,z) for that pixel, in the SAME world frame as " - "agentview world_NN.npy." + "only. The matching wrist world-map observation gives world " + "(x,y,z) for that pixel in the same world frame as the " + "agentview world-map artifact." + ) + env_state.save( + "wrist_metadata.json", + wmeta_out, + step=step_idx, ) - with open( - artifact_path(output_dir, "metadata", step=step_idx, camera="wrist", resolution="low"), - "w", - ) as f: - json.dump(wmeta_out, f, indent=2) except Exception as e: logger.warning("wrist depth/world dump failed: %s", e) @@ -1069,20 +1011,19 @@ def dump_state(primitives: LiberoPrimitives, output_dir: str, step_idx: int, meta_hi = primitives.env.get_camera_meta("agentview", 1024, 1024) if meta_hi is None: raise RuntimeError("agentview camera metadata missing") - imageio.imwrite( - artifact_path(output_dir, "image", step=step_idx, camera="agentview", resolution="high"), + env_state.save( + "agentview_high.png", np.asarray(rgb_hi)[::-1], + step=step_idx, ) world_hi = _world_from_depth( _metric_depth(depth_hi, meta_hi)[::-1], meta_hi, ).astype(np.float16) - np.save( - artifact_path(output_dir, "world", step=step_idx, camera="agentview", resolution="high"), + env_state.save( + "agentview_world_high.npy", world_hi, - ) - agent_world_map_hi = _artifact_relative_path( - step_idx, "agentview", "high", "world" + step=step_idx, ) except Exception as e: logger.warning("agentview high-res dump failed: %s", e) @@ -1099,86 +1040,35 @@ def dump_state(primitives: LiberoPrimitives, output_dir: str, step_idx: int, ) if meta_wrist_hi is None: raise RuntimeError("robot0_eye_in_hand camera metadata missing") - imageio.imwrite( - artifact_path(output_dir, "image", step=step_idx, camera="wrist", resolution="high"), + env_state.save( + "wrist_high.png", np.asarray(rgb_wrist_hi)[::-1], + step=step_idx, ) world_wrist_hi = _world_from_depth( _metric_depth(depth_wrist_hi, meta_wrist_hi)[::-1], meta_wrist_hi, ).astype(np.float16) - np.save( - artifact_path(output_dir, "world", step=step_idx, camera="wrist", resolution="high"), + env_state.save( + "wrist_world_high.npy", world_wrist_hi, - ) - wrist_world_map_hi = _artifact_relative_path( - step_idx, "wrist", "high", "world" + step=step_idx, ) except Exception as e: logger.warning("wrist high-res dump failed: %s", e) - for old_step in range(max(0, int(step_idx) - 4)): - for path in ( - artifact_path(output_dir, "image", step=old_step, camera="agentview", resolution="high"), - artifact_path(output_dir, "world", step=old_step, camera="agentview", resolution="high"), - artifact_path(output_dir, "image", step=old_step, camera="wrist", resolution="high"), - artifact_path(output_dir, "world", step=old_step, camera="wrist", resolution="high"), - ): - try: - os.unlink(path) - except FileNotFoundError: - pass - - blob = { - "step_idx": step_idx, - "libero_terminated": primitives.env.episode_terminated, - "episode_truncated": primitives.env.episode_truncated, - "task_language": primitives.env.get_task_language(), - "state": state, - "world_map": agent_world_map, - "wrist_world_map": wrist_world_map, - "world_map_hi": agent_world_map_hi, - "wrist_world_map_hi": wrist_world_map_hi, - } - # Merge the execution log (command + result + elapsed_s) into the - # state blob so a single entry captures everything for the step. - if log is not None: - blob["command"] = log.get("command") - blob["result"] = log.get("result") - blob["elapsed_s"] = log.get("elapsed_s") - _append_state(output_dir, blob) - return blob - # --------------------------------------------------------------------------- # Tool schema declarations (Anthropic-shaped canonical schema) # --------------------------------------------------------------------------- -PRIMITIVE_TOOL_NAMES: tuple[str, ...] = ( - "move_to", - "pi0_pick", - "pi0_doubled", - "release", - "set_gripper", - "rotate_wrist", - "rotate_pitch", - "move_pose", -) - TOOLS_SPEC = [ { - "name": "view_driver_state", + "name": "view_env_state", "description": ( - "Read step NN from `states.json` + the matching " - "state images in {{output_dir}}. If step is " - "null, returns the latest entry. Each entry contains the robot " - "state, libero_terminated flag, command log, and result. Returns " - "available PNG paths in this stable " - "order: 1) `images/image_NN.png` (Pi0-frame agentview), " - "2) `images_cam/image_cam_NN.png` (calibration-frame agentview), " - "3) `images_wrist/image_wrist_NN.png` (calibration-frame wrist). " - "High-resolution calibration-frame images are returned as file " - "paths, not embedded as image bytes. " + "Read one recorded state and its observation artifacts. Step -1 " + "selects the latest entry. Embeds policy, agentview, and wrist " + "images when available. " "Use the calibration-frame images for pixel back-projection; JSON " "state alone is not enough. Use agentview for global tabletop " "layout and object locations; use wrist for close-range details " @@ -1188,8 +1078,9 @@ def dump_state(primitives: LiberoPrimitives, output_dir: str, step_idx: int, "type": "object", "properties": { "step": { - "type": ["integer", "null"], - "description": "Step number; 0 = initial. Null = latest.", + "type": "integer", + "default": -1, + "description": "Step number; 0 = initial, -1 = latest.", }, }, }, @@ -1257,7 +1148,7 @@ def dump_state(primitives: LiberoPrimitives, output_dir: str, step_idx: int, "description": ( "Pi0.5 closed-loop contact skill for non-pick interactions " "(e.g. stove/knob/button/short push). Returned success/task_success " - "only mirrors official libero_terminated; for intermediate contact " + "only mirrors official termination; for intermediate contact " "skills, success=false does not necessarily mean the contact " "interaction failed. Inspect image/state evidence. Do not use it " "as a general pick/place shortcut." @@ -1380,9 +1271,7 @@ def dump_state(primitives: LiberoPrimitives, output_dir: str, step_idx: int, { "name": "view_camera_meta", "description": ( - "Read camera calibration metadata from the output dir. " - "camera='agentview' reads static camera_meta.json. " - "camera='wrist' reads the per-step wrist metadata." + "Read per-step camera calibration metadata from recorded artifacts." ), "input_schema": { "type": "object", @@ -1393,8 +1282,9 @@ def dump_state(primitives: LiberoPrimitives, output_dir: str, step_idx: int, "description": "Camera metadata to read (default agentview).", }, "step": { - "type": ["integer", "null"], - "description": "Wrist metadata step to use (default latest).", + "type": "integer", + "default": -1, + "description": "Metadata step to use; -1 = latest.", }, }, }, @@ -1420,8 +1310,9 @@ def dump_state(primitives: LiberoPrimitives, output_dir: str, step_idx: int, "description": "Artifact camera to use (default agentview).", }, "step": { - "type": ["integer", "null"], - "description": "Step NN to segment; null = latest.", + "type": "integer", + "default": -1, + "description": "Step to segment; -1 = latest.", }, "point": { "type": ["array", "null"], @@ -1447,7 +1338,7 @@ def dump_state(primitives: LiberoPrimitives, output_dir: str, step_idx: int, "selected camera's precomputed world map. Row 0 = top of image, " "col 0 = left. Returns world_xyz in meters.\n\n" "USE THIS to find where an object is in the world — look at " - "the high-resolution paths returned by view_driver_state " + "the embedded high-resolution image returned by view_env_state " "to pick a pixel on the target object, then call back_project. " "The default resolution is high (1024x1024). Pass " "resolution='low' only for pixels from the embedded/standard " @@ -1476,8 +1367,9 @@ def dump_state(primitives: LiberoPrimitives, output_dir: str, step_idx: int, "description": "Pixel column (0=left) in the selected resolution image.", }, "step": { - "type": ["integer", "null"], - "description": "Depth/world-map step to use (default latest). 0 for initial.", + "type": "integer", + "default": -1, + "description": "Depth/world-map step; 0 = initial, -1 = latest.", }, "camera": { "type": "string", @@ -1517,151 +1409,69 @@ def dump_state(primitives: LiberoPrimitives, output_dir: str, step_idx: int, ] -# --------------------------------------------------------------------------- -# State trace readers -# --------------------------------------------------------------------------- - - -def _load_states() -> list: - """Return the parsed state trace from the local output dir.""" - path = artifact_path(get_output_dir(), "states") - if not path.exists(): - return [] - with open(path) as f: - return json.load(f) - - -def _latest_step() -> int | None: - states = _load_states() - if not states: - return None - return states[-1]["step_idx"] - - -def _load_step(nn: int) -> dict: - """Look up the state blob for step ``nn`` from states.json.""" - for entry in _load_states(): - if entry.get("step_idx") == nn: - return entry - raise FileNotFoundError(f"step {nn} not present in states.json") - - -def _load_image_path(nn: int, kind: str) -> str | None: - """Return the path to a dumped state image. None if not present.""" - out_dir = get_output_dir() - if kind == "agent": - path = artifact_path(out_dir, "policy_image", step=nn, camera="agentview", resolution="low") - elif kind == "camera": - path = artifact_path(out_dir, "image", step=nn, camera="agentview", resolution="low") - elif kind == "wrist": - path = artifact_path(out_dir, "image", step=nn, camera="wrist", resolution="low") - else: - raise ValueError(f"unknown image kind: {kind}") - if not path.exists(): - return None - return str(path) - - -def _load_camera_meta(camera: str = "agentview", nn: int | None = None) -> dict: - out_dir = get_output_dir() - if camera == "agentview": - path = artifact_path(out_dir, "metadata", camera="agentview", resolution="low") - elif camera == "wrist" and nn is not None: - path = artifact_path(out_dir, "metadata", step=nn, camera="wrist", resolution="low") - else: - raise ValueError("camera must be 'agentview' or 'wrist' with nn") - if not path.exists(): - raise FileNotFoundError(f"{path.name} not found in {out_dir}") - with open(path) as f: - return json.load(f) - - -def _load_depth(camera: str, nn: int) -> np.ndarray: - out_dir = get_output_dir() - if camera not in ("agentview", "wrist"): - raise ValueError("camera must be 'agentview' or 'wrist'") - path = artifact_path(out_dir, "depth", step=nn, camera=camera, resolution="low") - if not path.exists(): - raise FileNotFoundError(f"{path.name} not found in {out_dir}") - depth = np.load(path) - if depth.ndim == 3: - depth = depth[..., 0] - return depth - - -def view_driver_state(step: int | None = None) -> dict: - latest = _latest_step() - if latest is None: - return {"error": "no state entries; env not ready"} - nn = latest if step is None else int(step) +@readonly +def view_env_state(step: int = -1, *, state: EnvState) -> dict: try: - data = _load_step(nn) - except Exception as e: - return {"error": f"step {nn} not present in state trace: {e}"} - - out: dict = {"step": nn} - out["task_language"] = data.get("task_language") - # Default to {} (not the entire blob) when "state" is missing — otherwise - # command/result/vla_desync would bleed into the "state" field, confusing - # the LLM about what robot state actually contains. - out["state"] = data.get("state", {}) - out["libero_terminated"] = data.get("libero_terminated") - out["episode_truncated"] = data.get("episode_truncated") - out["world_map"] = data.get("world_map") - out["wrist_world_map"] = data.get("wrist_world_map") - out["world_map_hi"] = data.get("world_map_hi") - out["wrist_world_map_hi"] = data.get("wrist_world_map_hi") + record = state.get(step) + except Exception as exc: + return {"error": f"state step not available: {exc}"} + + nn = record.step_idx + extras = record.extras + out: dict = { + "step": nn, + "terminated": record.terminated, + "truncated": record.truncated, + "state": record.state, + "artifacts": sorted(record.artifacts), + } + out["task_language"] = extras.get("task_language") out["log"] = { - "command": data.get("command"), - "result": data.get("result"), - "elapsed_s": data.get("elapsed_s"), + "command": record.command, + "result": record.result, + "elapsed_s": record.elapsed_s, } - for field, kind in ( - ("image_path", "agent"), - ("image_cam_path", "camera"), - ("image_wrist_path", "wrist"), - ): - image_path = _load_image_path(nn, kind) - if image_path: - out[field] = image_path - for field, camera in ( - ("image_cam_hi_path", "agentview"), - ("image_wrist_hi_path", "wrist"), + for slot, names in ( + ("_image_bytes", ("agentview_policy.png",)), + ("_image_cam_bytes", ("agentview_high.png", "agentview.png")), + ("_image_wrist_bytes", ("wrist_high.png", "wrist.png")), ): - image_path = artifact_path(get_output_dir(), "image", step=nn, camera=camera, resolution="high") - if image_path.exists(): - out[field] = str(image_path) + name = next((name for name in names if name in record.artifacts), None) + if name: + try: + out[slot] = state.load_bytes(name, step=nn) + except FileNotFoundError: + pass return out -def _select_segment_artifacts(nn: int, camera: str): - out_dir = get_output_dir() +def _select_segment_artifacts( + state: EnvState, + record: StepRecord, + camera: str, +) -> tuple[str | None, str | None, list[tuple[str | None, str | None]]]: if camera not in ("agentview", "wrist"): raise ValueError(f"unknown segment camera: {camera}") pairs = [ - ( - artifact_path(out_dir, "image", step=nn, camera=camera, resolution=resolution), - artifact_path(out_dir, "world", step=nn, camera=camera, resolution=resolution), - ) - for resolution in ("high", "low") + (f"{camera}_high.png", f"{camera}_world_high.npy"), + (f"{camera}.png", f"{camera}_world.npy"), ] - - for image_path, world_path in pairs: - if image_path.exists() and world_path.exists(): - return image_path, world_path, pairs + for image_name, world_name in pairs: + if ( + image_name in record.artifacts + and world_name in record.artifacts + and state.exists(image_name, step=record.step_idx) + and state.exists(world_name, step=record.step_idx) + ): + return image_name, world_name, pairs return None, None, pairs -def _next_segment_artifact_paths(out_dir: Path, nn: int): - segments_dir = artifact_path(out_dir, "segments") - segments_dir.mkdir(parents=True, exist_ok=True) +def _next_segment_index(record: StepRecord) -> int: idx = 0 - while True: - segment_path = segments_dir / f"segment_{nn:02d}_{idx:02d}.json" - overlay_path = segments_dir / f"segment_overlay_{nn:02d}_{idx:02d}.png" - if not segment_path.exists() and not overlay_path.exists(): - return segment_path, overlay_path, idx + while f"segment_{idx:02d}.json" in record.artifacts: idx += 1 + return idx def _mask_to_world(mask: np.ndarray, world_map: np.ndarray, @@ -1718,56 +1528,60 @@ def _mask_to_world(mask: np.ndarray, world_map: np.ndarray, return result -def _write_segment_overlay(image_path: Path, mask: np.ndarray, - overlay_path: Path) -> bool: - try: - image = imageio.imread(image_path) - if image.ndim != 3 or image.shape[:2] != mask.shape: - return False - overlay = image.copy() - red = np.zeros_like(overlay) - red[..., 0] = 255 - overlay[mask] = ( - 0.55 * overlay[mask].astype(np.float32) - + 0.45 * red[mask].astype(np.float32) - ).astype(np.uint8) - imageio.imwrite(overlay_path, overlay) - return overlay_path.exists() - except Exception: - return False - - -def view_camera_meta(camera: str = "agentview", step: int | None = None) -> dict: +def _make_segment_overlay( + image: np.ndarray, + mask: np.ndarray, +) -> np.ndarray | None: + if image.ndim != 3 or image.shape[:2] != mask.shape: + return None + overlay = image.copy() + red = np.zeros_like(overlay) + red[..., 0] = 255 + overlay[mask] = ( + 0.55 * overlay[mask].astype(np.float32) + + 0.45 * red[mask].astype(np.float32) + ).astype(np.uint8) + return overlay + + +@readonly +def view_camera_meta( + camera: str = "agentview", + step: int = -1, + *, + state: EnvState, +) -> dict: """Read camera calibration metadata for localization.""" if camera not in ("agentview", "wrist"): return {"error": f"bad camera '{camera}' (use 'agentview' or 'wrist')"} - nn = None - if camera == "wrist": - nn = _latest_step() if step is None else int(step) - if nn is None: - return {"error": "no wrist metadata available"} - try: - meta = _load_camera_meta(camera, nn) + record = state.get(step) + metadata_name = f"{camera}_metadata.json" + if metadata_name not in record.artifacts: + raise FileNotFoundError(metadata_name) + meta = state.load(metadata_name, step=record.step_idx) except Exception as e: return {"error": f"{camera} camera metadata not found: {e}"} if camera == "agentview": return {"camera": "agentview", "camera_meta": meta} - return {"camera": "wrist", "step": nn, "camera_meta": meta} + return {"camera": "wrist", "step": record.step_idx, "camera_meta": meta} +@readonly def back_project( row: int | None = None, col: int | None = None, - step: int | None = None, + step: int = -1, camera: str = "agentview", resolution: str = "high", row_range: list | None = None, col_range: list | None = None, z_min: float | None = None, z_max: float | None = None, + *, + state: EnvState, ) -> dict: """Look up a pixel's world XYZ in the precomputed world map.""" if camera not in ("agentview", "wrist"): @@ -1784,24 +1598,16 @@ def back_project( ) } - latest = _latest_step() - nn = latest if step is None else int(step) - if nn is None: - return {"error": "no depth/world-map files available"} - try: - data = _load_step(nn) + record = state.get(step) except Exception as e: - return {"error": f"step {nn} not present in state trace: {e}"} + return {"error": f"state step not available: {e}"} + nn = record.step_idx - if camera == "agentview": - hi_artifact = data.get("world_map_hi") - low_artifact = data.get("world_map") - else: - hi_artifact = data.get("wrist_world_map_hi") - low_artifact = data.get("wrist_world_map") + hi_artifact = f"{camera}_world_high.npy" + low_artifact = f"{camera}_world.npy" source_artifact = hi_artifact if resolution == "high" else low_artifact - if not source_artifact: + if source_artifact not in record.artifacts: return { "error": ( f"{camera} {resolution}-resolution world map not recorded " @@ -1810,8 +1616,7 @@ def back_project( } try: - world_path = artifact_path(get_output_dir(), "world", step=nn, camera=camera, resolution=resolution) - world_map = np.load(world_path) + world_map = state.load(str(source_artifact), step=nn) except Exception as e: return { "error": ( @@ -1897,7 +1702,12 @@ def back_project( depth_m = None if source_artifact == low_artifact: try: - depth = _load_depth(camera, nn) + depth_artifact = f"{camera}_depth.npy" + if depth_artifact not in record.artifacts: + raise FileNotFoundError(depth_artifact) + depth = state.load(depth_artifact, step=nn) + if depth.ndim == 3: + depth = depth[..., 0] except Exception as e: return {"error": f"{camera} depth not found for step {nn}: {e}"} depth_m = float(depth[row, col]) diff --git a/rpent/cli/dashboard.py b/rpent/cli/dashboard.py index c63e5c49..9a5f324c 100644 --- a/rpent/cli/dashboard.py +++ b/rpent/cli/dashboard.py @@ -155,7 +155,6 @@ def _run_dashboard_task( toolkit = get_toolkit( args.env_name, primitives_kwargs=primitives_kwargs, - video_path=str(output_dir / "episode.mp4"), dashboard_events=state, ) planner = build_planner( diff --git a/rpent/cli/main.py b/rpent/cli/main.py index 889ea17b..94b22f1b 100644 --- a/rpent/cli/main.py +++ b/rpent/cli/main.py @@ -37,7 +37,7 @@ NullDashboardEventSink, RunStartedEvent, ) -from rpent.envs import get_env_spec, get_toolkit +from rpent.envs import enumerate_envs, get_env_spec, get_toolkit from rpent.planner.base import build_planner from rpent.utils.logging import get_logger, init_output_dir from rpent.utils.resources import ensure_resources @@ -82,12 +82,19 @@ def _serialize_messages(messages: list[dict]) -> list[dict]: def _build_argparser() -> argparse.ArgumentParser: + known_envs = enumerate_envs() + known_envs_text = ", ".join(known_envs) if known_envs else "none" ap = argparse.ArgumentParser( - description="Standalone hybrid LLM-in-the-loop physical agent", + description="RPent: Agentic Infrastructure for the Physical World", ) - ap.add_argument("--env", dest="env_name", required=True, choices=["libero"], - help="Environment backend: libero.") + ap.add_argument( + "--env", + dest="env_name", + required=True, + choices=known_envs, + help=f"Environment backend. Known environments: {known_envs_text}.", + ) # models ap.add_argument("--planner", default="api", @@ -106,7 +113,7 @@ def _build_argparser() -> argparse.ArgumentParser: help="Never send image bytes to the model (api planner only). " "Use for text-only models that reject image input " "(e.g. 400 \"message type 'image_url' is not supported\"); " - "read_image then returns the file path with a notice.") + "read_image then returns the image name instead, with a notice.") ap.add_argument("--planner-timeout-s", type=int, default=None, help="Wall-clock cap for api/claude_code/codex planner runs. " "Terminal interactive API/Claude sessions are exempt. " @@ -220,7 +227,6 @@ def main() -> int: toolkit = get_toolkit( env_name, primitives_kwargs=primitives_kwargs, - video_path=str(Path(output_dir) / "episode.mp4"), dashboard_events=dashboard_events, ) diff --git a/rpent/context/prompts/prompt.py b/rpent/context/prompts/prompt.py index b3c9e47b..43e6dad0 100644 --- a/rpent/context/prompts/prompt.py +++ b/rpent/context/prompts/prompt.py @@ -5,7 +5,7 @@ OUTPUT = BulletList([ "Brief reasoning before each tool call (1-2 sentences): observation -> decision.", - "Don't re-read files already in this session. Don't view_driver_state right after a primitive tool already returned the state.", + "Don't re-read files already in this session. Don't view_env_state right after a primitive tool already returned the state.", "Numerical coords in 3 decimals are enough.", "Save artifacts BEFORE calling finish. Stop immediately after writing the audit; do not chat further.", ]) diff --git a/rpent/dashboard/events.py b/rpent/dashboard/events.py index 5784624d..0c6cbfae 100644 --- a/rpent/dashboard/events.py +++ b/rpent/dashboard/events.py @@ -39,6 +39,15 @@ class ToolResultEvent: result: Any +@dataclass(frozen=True, slots=True) +class StepRecordEvent: + """Publish one recorded environment step and its artifact context.""" + + record: Any + env_state: Any + frame_artifacts: dict[str, str] + + @dataclass(frozen=True, slots=True) class RunStartedEvent: """Mark startup complete and the agent run active.""" @@ -49,6 +58,7 @@ class RunStartedEvent: | UsageEvent | RuntimeStatusEvent | ToolResultEvent + | StepRecordEvent | RunStartedEvent ) diff --git a/rpent/dashboard/state.py b/rpent/dashboard/state.py index ae495829..32cebabf 100644 --- a/rpent/dashboard/state.py +++ b/rpent/dashboard/state.py @@ -7,12 +7,13 @@ import uuid from dataclasses import dataclass, replace from pathlib import Path -from typing import Any, Literal +from typing import TYPE_CHECKING, Any, Literal from rpent.dashboard.events import ( DashboardEvent, RunStartedEvent, RuntimeStatusEvent, + StepRecordEvent, ToolResultEvent, TranscriptEvent, UsageEvent, @@ -26,6 +27,9 @@ UnknownDashboardMessageError, ) +if TYPE_CHECKING: + from rpent.tools.state import EnvState, StepRecord + RUNTIME_STATUSES = {"pending", "starting", "ready", "failed"} TERMINAL_RUN_STATES = {"succeeded", "failed", "cancelled"} _PLANNER_ACTIVITIES = {"starting", "idle", "busy", "ended"} @@ -111,6 +115,7 @@ def __init__( self._condition = threading.Condition(self._lock) self._task_state: str | None = None self._terminated = False + self._truncated = False self._error: str | None = None self._usage = {"in": 0, "out": 0, "tool_calls": 0} self._runtime = { @@ -121,6 +126,8 @@ def __init__( self._timeline: list[dict[str, Any]] = [] self._frames: dict[str, bytes] = {} self._frame_idx = -1 + self.env_state: EnvState | None = None + self.frame_artifacts: dict[str, str] = {} self._accepting_input = False self._planner_activity: PlannerActivity = "starting" self._interrupt_requested = False @@ -274,7 +281,12 @@ def complete_task( raise ValueError(f"invalid terminal run state: {state!r}") with self._condition: self._task_state = state - self._terminated = any(item.get("terminated") for item in self._timeline) + self._terminated = any( + item.get("terminated") for item in self._timeline + ) + self._truncated = any( + item.get("truncated") for item in self._timeline + ) self._error = None if error is None else str(error) self._task_replacement_requested = False self._seal_interaction_locked() @@ -300,6 +312,7 @@ def _begin_task_locked( self._task_state = "starting" self._task_replacement_requested = False self._terminated = False + self._truncated = False self._error = None self._usage = {"in": 0, "out": 0, "tool_calls": 0} self._reset_task_runtime_locked() @@ -307,6 +320,8 @@ def _begin_task_locked( self._timeline = [] self._frames = {} self._frame_idx = -1 + self.env_state = None + self.frame_artifacts = {} self._accepting_input = False self._planner_activity = "starting" self._interrupt_requested = False @@ -538,6 +553,11 @@ def emit(self, event: DashboardEvent) -> None: if isinstance(event, ToolResultEvent): self._apply_tool_result(event) return + if isinstance(event, StepRecordEvent): + self.env_state = event.env_state + self.frame_artifacts = dict(event.frame_artifacts) + self.on_step(event.record) + return if isinstance(event, RunStartedEvent): self._start() return @@ -572,7 +592,14 @@ def _apply_tool_result(self, event: ToolResultEvent) -> None: result = event.result if not isinstance(result, dict): return - self._apply_frame_paths(result) + frames = { + "camera": result.get("_image_cam_bytes") or result.get("_image_bytes"), + "wrist": result.get("_image_wrist_bytes"), + } + self._update_frames( + step=result.get("step"), + frames={kind: data for kind, data in frames.items() if data}, + ) log = result.get("log") if not isinstance(log, dict): return @@ -584,15 +611,15 @@ def _apply_tool_result(self, event: ToolResultEvent) -> None: except Exception: return action = str(command.get("action", name)) - terminated = bool( - result.get("terminated", result.get("libero_terminated", False)) - ) + terminated = bool(result.get("terminated")) + truncated = bool(result.get("truncated")) action_video_path = self._action_video_from_result( result, step=step, action=action, ) action_video = str(action_video_path) if action_video_path is not None else None + action_video_artifact = result.get("action_video_artifact") item = { "step": step, "action": action, @@ -600,12 +627,15 @@ def _apply_tool_result(self, event: ToolResultEvent) -> None: "result": log.get("result"), "elapsed_s": log.get("elapsed_s"), "terminated": terminated, - "has_action_video": action_video_path is not None, + "truncated": truncated, "action_video_path": action_video, + "action_video_artifact": action_video_artifact, + "has_action_video": bool(action_video_artifact or action_video_path), } with self._lock: self._timeline.append(item) self._terminated = self._terminated or terminated + self._truncated = self._truncated or truncated def _action_video_from_result( self, @@ -623,6 +653,47 @@ def _action_video_from_result( return None return path if path.exists() else None + def on_step(self, record: StepRecord) -> None: + """Project one recorded environment step into frames and timeline.""" + self._update_step_frames(record) + command = record.command + if not isinstance(command, dict) or not command.get("action"): + return + action_video = next( + (name for name in sorted(record.artifacts) if name.endswith(".mp4")), + None, + ) + item = { + "step": record.step_idx, + "action": str(command.get("action")), + "args": {key: value for key, value in command.items() if key != "action"}, + "result": record.result, + "elapsed_s": record.elapsed_s, + "terminated": record.terminated, + "truncated": record.truncated, + "action_video_artifact": action_video, + "has_action_video": action_video is not None, + } + with self._lock: + self._timeline.append(item) + self._terminated = self._terminated or record.terminated + self._truncated = self._truncated or record.truncated + + def _update_step_frames(self, record: StepRecord) -> None: + """Load dashboard frame bytes from the step's canonical artifacts.""" + env_state = self.env_state + if env_state is None: + return + frames: dict[str, bytes] = {} + for kind, artifact in self.frame_artifacts.items(): + if kind not in self._frame_names or artifact not in record.artifacts: + continue + try: + frames[kind] = env_state.load_bytes(artifact, step=record.step_idx) + except FileNotFoundError: + continue + self._update_frames(step=record.step_idx, frames=frames) + def _resolve_output_path(self, value: Any) -> Path: path = Path(value) if path.is_absolute() or path.is_relative_to(self.output_dir): @@ -774,15 +845,25 @@ def frame(self, kind: str) -> bytes | None: return self._frames.get(kind) def action_video_path(self, step: int) -> Path | None: + env_state = self.env_state with self._lock: + artifact = None + raw_path = None for item in self._timeline: if int(item.get("step", -1)) != int(step): continue + artifact = item.get("action_video_artifact") raw_path = item.get("action_video_path") - if not raw_path: - return None - video_path = Path(raw_path) - return video_path if video_path.exists() else None + break + if artifact and env_state is not None: + try: + path = env_state.artifact_path(artifact, step=int(step)) + except (LookupError, ValueError): + return None + return path if path.exists() else None + if raw_path: + video_path = Path(raw_path) + return video_path if video_path.exists() else None return None def has_video(self) -> bool: @@ -802,6 +883,7 @@ def snapshot(self) -> dict[str, Any]: return { "state": self._visible_state_locked(), "terminated": self._terminated, + "truncated": self._truncated, "error": self._error, "usage": dict(self._usage), "runtime": self._runtime_snapshot(), @@ -824,6 +906,7 @@ def run_detail(self) -> dict[str, Any]: return { "state": self._visible_state_locked(), "terminated": self._terminated, + "truncated": self._truncated, "error": self._error, "usage": dict(self._usage), "runtime": self._runtime_snapshot(), diff --git a/rpent/envs/__init__.py b/rpent/envs/__init__.py index f2720d34..7bfc9f0c 100644 --- a/rpent/envs/__init__.py +++ b/rpent/envs/__init__.py @@ -1,13 +1,14 @@ """Environment-specific RPent extensions.""" +from rpent.envs.base import enumerate_envs, get_env_spec, get_toolkit from rpent.envs.env_spec import EnvSpec, RunConfig from rpent.envs.prompt_bundle import PromptBundle -from rpent.envs.base import get_env_spec, get_toolkit __all__ = [ "EnvSpec", "PromptBundle", "RunConfig", + "enumerate_envs", "get_env_spec", "get_toolkit", ] diff --git a/rpent/envs/base.py b/rpent/envs/base.py index dfe5758d..9c8e4b91 100644 --- a/rpent/envs/base.py +++ b/rpent/envs/base.py @@ -10,6 +10,7 @@ from __future__ import annotations import importlib +import pkgutil import sys from typing import Any @@ -36,6 +37,17 @@ def _resolve_env(name: str) -> Any: raise ValueError(f"unknown env: {env_name!r}") from e +def enumerate_envs() -> tuple[str, ...]: + """Return the names of importable environment packages under ``robots``.""" + import robots + + return tuple(sorted( + module.name + for module in pkgutil.iter_modules(robots.__path__) + if module.ispkg and not module.name.startswith("_") + )) + + def get_env_spec(name: str) -> EnvSpec: return _resolve_env(name).get_env_spec() diff --git a/rpent/planner/api_loop.py b/rpent/planner/api_loop.py index f7a9c106..2a96a43d 100644 --- a/rpent/planner/api_loop.py +++ b/rpent/planner/api_loop.py @@ -15,6 +15,8 @@ import json import queue from collections import deque +from collections.abc import Callable +from pathlib import Path from typing import Any from pydantic_ai import Agent, BinaryContent, ModelSettings, Tool, ToolReturn @@ -42,6 +44,7 @@ from rpent.dashboard.interaction import DashboardInteractionPort, DashboardMessage from rpent.dashboard.planner_control import DashboardPlannerControl from rpent.planner.base import PlannerResult +from rpent.tools.state import EnvState from rpent.tools.toolkit import Toolkit from rpent.utils.logging import get_logger @@ -689,7 +692,7 @@ def _api_error_text(error: Exception, *, no_images: bool) -> str: def _build_tools(toolkit: Toolkit, *, no_images: bool = False) -> list[Tool]: """Build the API-only image reader plus pydantic-ai toolkit wrappers.""" - image_reader = read_image_text_only if no_images else read_image + image_reader = _make_image_reader(toolkit.state, no_images=no_images) tools: list[Tool] = [Tool(image_reader, name="read_image")] for spec in toolkit.get_tools_spec(): name = spec["name"] @@ -706,23 +709,72 @@ def _build_tools(toolkit: Toolkit, *, no_images: bool = False) -> list[Tool]: return tools -def read_image(path: str) -> ToolReturn: - """Read a local image path returned by an RPent tool as visual input.""" +def _make_image_reader( + state: EnvState, + *, + no_images: bool, +) -> Callable[[str, int], ToolReturn | str]: + if no_images: + + def read_image_tool(name: str, step: int = -1) -> str: + return read_image_text_only(name, step, state=state) + + read_image_tool.__name__ = "read_image" + read_image_tool.__doc__ = read_image_text_only.__doc__ + return read_image_tool + + def read_image_tool(name: str, step: int = -1) -> ToolReturn: + return read_image(name, step, state=state) + + read_image_tool.__name__ = "read_image" + read_image_tool.__doc__ = read_image.__doc__ + return read_image_tool + + +def read_image(name: str, step: int = -1, *, state: EnvState) -> ToolReturn: + """Read a step-scoped image artifact as visual input.""" + resolved_step, path = _resolve_image_artifact(state, name, step) return ToolReturn( - return_value=path, - content=[BinaryContent.from_path(path)], + return_value={"artifact": name, "step": resolved_step}, + content=[ + BinaryContent( + data=state.load_bytes(name, step=resolved_step), + media_type=_image_media_type(path), + ) + ], ) -def read_image_text_only(path: str) -> str: - """``read_image`` stub for ``--no-images``: acknowledge, send no bytes.""" +def read_image_text_only(name: str, step: int = -1, *, state: EnvState) -> str: + """Acknowledge an image artifact without sending bytes to the model.""" + resolved_step, _ = _resolve_image_artifact(state, name, step) return ( - f"{path} exists, but image input is disabled (--no-images, text-only " - "model). Reason from textual state instead: view_driver_state, " - "back_project, and the numeric fields in tool results." + f"Image artifact {name!r} exists at step {resolved_step}, but image " + "input is disabled (--no-images, text-only model). Reason from textual " + "state instead: view_env_state, back_project, and numeric tool results." ) +def _resolve_image_artifact( + state: EnvState, + name: str, + step: int, +) -> tuple[int, Path]: + record = state.get(step) + path = state.artifact_path(name, step=record.step_idx) + if name not in record.artifacts or not path.is_file(): + raise FileNotFoundError( + f"image artifact {name!r} is not available at step {step}" + ) + if path.suffix.lower() not in {".png", ".jpg", ".jpeg"}: + raise ValueError(f"artifact {name!r} is not an image") + return record.step_idx, path + + +def _image_media_type(path: Path) -> str: + return "image/jpeg" if path.suffix.lower() in {".jpg", ".jpeg"} else "image/png" + + def _make_tool_function(toolkit: Toolkit, name: str, *, no_images: bool = False): """Return a callable that dispatches one tool call to the toolkit.""" diff --git a/rpent/tools/common.py b/rpent/tools/common.py index fd959566..be790214 100644 --- a/rpent/tools/common.py +++ b/rpent/tools/common.py @@ -4,6 +4,7 @@ import os from pathlib import Path +from rpent.tools.toolkit import readonly from rpent.utils.config import get_repo_root from rpent.utils.logging import get_output_dir @@ -93,6 +94,7 @@ def _truncate(text: str, max_chars: int) -> str: ) +@readonly def read_text_file(path: str, max_chars: int = 40000) -> dict: p = _resolve(path) if not p.exists(): @@ -106,6 +108,7 @@ def read_text_file(path: str, max_chars: int = 40000) -> dict: return {"path": str(p), "size": len(text), "content": _truncate(text, max_chars)} +@readonly def write_text_file(path: str, content: str) -> dict: p = _resolve(path) p.parent.mkdir(parents=True, exist_ok=True) @@ -113,6 +116,7 @@ def write_text_file(path: str, content: str) -> dict: return {"path": str(p), "bytes_written": len(content.encode("utf-8"))} +@readonly def list_dir(path: str = "") -> dict: # Default to the current output dir (so parallel agents see their own). p = _resolve(path) if path else get_output_dir() @@ -122,6 +126,7 @@ def list_dir(path: str = "") -> dict: return {"path": str(p), "count": len(files), "files": files} +@readonly def finish(status: str, summary: str) -> dict: """Signal that the run is complete. Halts the agent loop. diff --git a/rpent/tools/state.py b/rpent/tools/state.py new file mode 100644 index 00000000..b6feb499 --- /dev/null +++ b/rpent/tools/state.py @@ -0,0 +1,334 @@ +"""Per-run environment state and artifact storage.""" +from __future__ import annotations + +import copy +import json +import os +from collections.abc import Generator +from contextlib import contextmanager +from dataclasses import dataclass, field +from pathlib import Path +from typing import Any + +import imageio.v2 as imageio +import numpy as np + +from rpent.utils.logging import get_logger + +logger = get_logger("env_state") + +_MANIFEST_NAME = "states.json" +_IMAGE_SUFFIXES = {".png", ".jpg", ".jpeg"} +_TEXT_SUFFIXES = {".txt", ".md"} +_SUPPORTED_SUFFIXES = _IMAGE_SUFFIXES | { + ".npy", + ".json", + ".jsonl", + ".mp4", + ".bin", +} | _TEXT_SUFFIXES + + +def _json_default(value: Any) -> Any: + if isinstance(value, np.ndarray): + return value.tolist() + if isinstance(value, np.generic): + return value.item() + if isinstance(value, Path): + return str(value) + raise TypeError(f"{type(value).__name__} is not JSON serializable") + + +@dataclass +class StepRecord: + """One motion step and the artifact base names captured for it.""" + + step_idx: int + state: dict[str, Any] + terminated: bool = False + truncated: bool = False + artifacts: set[str] = field(default_factory=set) + command: dict | None = None + result: dict | None = None + elapsed_s: float | None = None + extras: dict[str, Any] = field(default_factory=dict) + + def to_blob(self) -> dict[str, Any]: + blob: dict[str, Any] = { + "step_idx": self.step_idx, + "state": self.state, + "terminated": self.terminated, + "truncated": self.truncated, + "artifacts": sorted(self.artifacts), + } + if self.command is not None: + blob["command"] = self.command + if self.result is not None: + blob["result"] = self.result + if self.elapsed_s is not None: + blob["elapsed_s"] = self.elapsed_s + if self.extras: + blob["extras"] = self.extras + return blob + +class EnvState: + """Own a run's step trace and all state-related files in its output root.""" + + def __init__(self, output_dir: Path | str): + self._output_dir = Path(output_dir) + self.reset() + + # -- private file resolution ----------------------------------------- + + @staticmethod + def _validate_name(name: str) -> str: + if not isinstance(name, str) or not name: + raise ValueError("artifact name must be a non-empty string") + path = Path(name) + if path.name != name or path.is_absolute() or name in {".", ".."}: + raise ValueError(f"artifact name must be a base filename: {name!r}") + if name == _MANIFEST_NAME: + raise ValueError(f"{_MANIFEST_NAME!r} is reserved") + if path.suffix.lower() not in _SUPPORTED_SUFFIXES: + raise ValueError(f"unsupported artifact suffix: {path.suffix or ''}") + return name + + def _artifact_file(self, name: str, step: int | None) -> Path: + name = self._validate_name(name) + if step is None: + return self._output_dir / name + if step < 0: + raise ValueError("artifact writes require a nonnegative step") + artifact = Path(name) + return self._output_dir / name / f"{step:02d}{artifact.suffix}" + + def _manifest_file(self) -> Path: + return self._output_dir / _MANIFEST_NAME + + def _temporary_file(self, destination: Path) -> Path: + return destination.with_name( + f".{destination.stem}.tmp{destination.suffix}" + ) + + def _record_for(self, step: int) -> StepRecord: + """Return the live step record at ``step`` for in-place updates.""" + if 0 <= step < len(self._steps): + return self._steps[step] + raise KeyError(f"step {step} not present in state trace") + + # -- manifest -------------------------------------------------------- + + def _write_manifest(self) -> None: + destination = self._manifest_file() + temporary = self._temporary_file(destination) + manifest = { + "run_artifacts": sorted(self._run_artifacts), + "steps": [record.to_blob() for record in self._steps], + } + try: + with temporary.open("w") as file: + json.dump(manifest, file, indent=2, default=_json_default) + os.replace(temporary, destination) + finally: + temporary.unlink(missing_ok=True) + + # -- lifecycle and counters ----------------------------------------- + + def reset(self) -> None: + """Reset the in-memory trace without removing on-disk artifacts.""" + self._output_dir.mkdir(parents=True, exist_ok=True) + self._steps: list[StepRecord] = [] + self._run_artifacts: set[str] = set() + self._step_open = False + + @property + def latest_step(self) -> int | None: + if not self._steps: + return None + return self._steps[-1].step_idx + + def latest_record(self) -> StepRecord | None: + """Return the most recently recorded step (live reference, no copy).""" + return self._steps[-1] if self._steps else None + + def _resolve_read_step(self, step: int | None) -> int | None: + if step is None: + return None + if step == -1: + latest = self.latest_step + if latest is None: + raise LookupError("no steps available") + return latest + if step < 0: + raise ValueError("step must be -1, None, or nonnegative") + return step + + # -- generic artifact operations ------------------------------------ + + def save( + self, + name: str, + value: Any, + *, + step: int | None = -1, + **options: Any, + ) -> str | None: + """Serialize ``value`` according to ``name`` and return its base name. + + ``step`` defaults to ``-1`` (the most recently recorded step), so calls + made inside (or right after) a :meth:`record_step` block attach to that + step without an explicit index. Pass an ``int`` to target a specific + step, or ``None`` for a run-level artifact such as an episode video. + """ + step = self._resolve_read_step(step) + destination = self._artifact_file(name, step) + destination.parent.mkdir(parents=True, exist_ok=True) + temporary = self._temporary_file(destination) + suffix = destination.suffix.lower() + record: StepRecord | None = ( + self._record_for(step) if step is not None else None + ) + try: + if suffix in _IMAGE_SUFFIXES: + array = np.asarray(value) + if array.dtype != np.uint8: + array = array.astype(np.uint8) + imageio.imwrite(temporary, array) + elif suffix == ".npy": + np.save(temporary, np.asarray(value)) + elif suffix == ".json": + with temporary.open("w") as file: + json.dump(value, file, indent=2, default=_json_default) + elif suffix == ".jsonl": + with temporary.open("w") as file: + if isinstance(value, str): + file.write(value) + else: + for item in value: + file.write( + json.dumps(item, default=_json_default) + "\n" + ) + elif suffix == ".mp4": + if isinstance(value, (bytes, bytearray, memoryview)): + temporary.write_bytes(bytes(value)) + else: + imageio.mimwrite( + temporary, + list(value), + fps=int(options.get("fps", 20)), + ) + elif suffix in _TEXT_SUFFIXES: + temporary.write_text(str(value)) + elif suffix == ".bin": + temporary.write_bytes(bytes(value)) + else: + raise ValueError(f"unsupported artifact suffix: {suffix}") + os.replace(temporary, destination) + if record is not None: + record.artifacts.add(name) + else: + self._run_artifacts.add(name) + self._write_manifest() + return name + except Exception as exc: + logger.warning("failed to save artifact %s: %s", name, exc) + return None + finally: + temporary.unlink(missing_ok=True) + + def load(self, name: str, *, step: int | None = -1) -> Any: + """Load an artifact; ``step=-1`` selects the latest recorded step.""" + resolved_step = self._resolve_read_step(step) + source = self._artifact_file(name, resolved_step) + suffix = source.suffix.lower() + if suffix in _IMAGE_SUFFIXES: + return imageio.imread(source) + if suffix == ".npy": + return np.load(source) + if suffix == ".json": + with source.open() as file: + return json.load(file) + if suffix == ".jsonl": + with source.open() as file: + return [json.loads(line) for line in file if line.strip()] + if suffix in _TEXT_SUFFIXES: + return source.read_text() + return source.read_bytes() + + def load_bytes(self, name: str, *, step: int | None = -1) -> bytes: + resolved_step = self._resolve_read_step(step) + return self._artifact_file(name, resolved_step).read_bytes() + + def artifact_path(self, name: str, *, step: int | None = -1) -> Path: + """Return the canonical filesystem path for an artifact.""" + resolved_step = self._resolve_read_step(step) + return self._artifact_file(name, resolved_step) + + def exists(self, name: str, *, step: int | None = -1) -> bool: + try: + resolved_step = self._resolve_read_step(step) + except LookupError: + return False + return self._artifact_file(name, resolved_step).exists() + + # -- step records ---------------------------------------------------- + + @contextmanager + def record_step( + self, + *, + state: dict[str, Any], + terminated: bool = False, + truncated: bool = False, + command: dict | None = None, + result: dict | None = None, + elapsed_s: float | None = None, + extras: dict[str, Any] | None = None, + ) -> Generator[int, None, None]: + """Append a new step record and yield its index. + + The step is committed to the trace immediately, so any subsequent + :meth:`save` without an explicit ``step`` attaches to it. If the block + raises, the step and any artifacts written to it are rolled back. + """ + if self._step_open: + raise RuntimeError("a step record is already open") + record = StepRecord( + step_idx=len(self._steps), + state=copy.deepcopy(state), + terminated=terminated, + truncated=truncated, + command=copy.deepcopy(command), + result=copy.deepcopy(result), + elapsed_s=elapsed_s, + extras=copy.deepcopy(extras or {}), + ) + self._steps.append(record) + self._step_open = True + self._write_manifest() + try: + yield record.step_idx + except BaseException: + for name in list(record.artifacts): + destination = self._artifact_file(name, record.step_idx) + destination.unlink(missing_ok=True) + if self._steps and self._steps[-1] is record: + self._steps.pop() + try: + self._write_manifest() + except Exception as exc: + logger.warning( + "failed to rewrite manifest after step rollback: %s", exc + ) + raise + finally: + self._step_open = False + + def get(self, step: int = -1) -> StepRecord: + resolved_step = self._resolve_read_step(step) + if resolved_step is None: + raise ValueError("step records are not run-level artifacts") + return copy.deepcopy(self._record_for(resolved_step)) + + def records(self) -> list[StepRecord]: + return copy.deepcopy(self._steps) diff --git a/rpent/tools/toolkit.py b/rpent/tools/toolkit.py index 31a7cbca..690f60af 100644 --- a/rpent/tools/toolkit.py +++ b/rpent/tools/toolkit.py @@ -9,14 +9,19 @@ import base64 import json import threading +import time import traceback from collections.abc import Callable from dataclasses import dataclass, field -from typing import Any, ClassVar +from functools import partial +from typing import TYPE_CHECKING, Any, ClassVar -from rpent.dashboard.events import DashboardEventSink, ToolResultEvent +from rpent.dashboard.events import DashboardEventSink, StepRecordEvent from rpent.utils.templates import substitute +if TYPE_CHECKING: + from rpent.tools.state import EnvState, StepRecord + @dataclass(slots=True) class _ToolOperation: @@ -28,6 +33,26 @@ class ToolCancelled(Exception): """Raised when an environment reaches a safe cancellation boundary.""" +def readonly(func): + """Mark a tool handler as not advancing environment state. + + Tool handlers capture a fresh observation (:meth:`Toolkit.get_env_state`) + by default. Apply this marker to observational and file/IO tools that do + not move the robot or otherwise change the environment. + """ + func._readonly = True + return func + + +def _is_readonly(handler: Callable[..., Any]) -> bool: + """Whether ``handler`` was marked with :func:`readonly`.""" + target = handler + while isinstance(target, partial): + target = target.func + target = getattr(target, "__func__", target) + return bool(getattr(target, "_readonly", False)) + + @dataclass class ToolResult: """Result of executing one tool call. @@ -106,10 +131,18 @@ class Toolkit: :meth:`close` to release env-side primitives / servers at the end of the run. """ - def __init__(self, *, dashboard_events: DashboardEventSink) -> None: - # name -> (spec, handler) - self._tools: dict[str, tuple[dict[str, Any], Callable[..., dict[str, Any]]]] = {} + def __init__( + self, + *, + dashboard_events: DashboardEventSink, + state: Any = None, + ) -> None: + self._tools: dict[ + str, + tuple[dict[str, Any], Callable[..., Any]], + ] = {} self._dashboard_events = dashboard_events + self._state = state self._operation_lock = threading.Lock() self._active_operation: _ToolOperation | None = None self._register_common_tools() @@ -122,7 +155,7 @@ def add_tool( self, name: str, spec: dict[str, Any], - handler: Callable[..., dict[str, Any]], + handler: Callable[..., Any], ) -> None: """Register one tool under ``name`` with its schema and handler. @@ -131,7 +164,8 @@ def add_tool( spec: Anthropic-shaped tool schema dict (``name``, ``description``, ``input_schema``). handler: Callable invoked with the tool's input kwargs; returns - a result dict. + a result dict. Decorate read-only handlers with + :func:`readonly`; all other handlers capture state. """ self._tools[name] = (spec, handler) @@ -147,6 +181,13 @@ def _register_common_tools(self) -> None: # Planner-facing API # ------------------------------------------------------------------ + @property + def state(self) -> EnvState: + """Return the run's artifact and step store.""" + if self._state is None: + raise RuntimeError("toolkit has no environment state") + return self._state + def get_tools_spec(self) -> list[dict[str, Any]]: """Return the tool schemas the LLM sees.""" return substitute( @@ -158,7 +199,7 @@ def execute_tool(self, name: str, input_dict: dict[str, Any]) -> ToolResult: entry = self._tools.get(name) if entry is None: return ToolResult(name=name, result={"error": f"unknown tool: {name}"}) - handler = entry[1] + _, handler = entry with self._operation_lock: if self._active_operation is not None: @@ -170,19 +211,80 @@ def execute_tool(self, name: str, input_dict: dict[str, Any]) -> ToolResult: self._active_operation = operation try: + started = time.perf_counter() + failed = False try: result = handler(**input_dict) except TypeError as e: - result = {"error": f"bad arguments for {name}: {e}", "got": input_dict} + result = { + "error": f"bad arguments for {name}: {e}", + "got": input_dict, + } + failed = True + except ToolCancelled as e: + result = { + "error": str(e), + "code": "tool_cancelled", + "interrupted": True, + } + failed = True except Exception as e: result = {"error": str(e), "traceback": traceback.format_exc()} - self._dashboard_events.emit(ToolResultEvent(name=name, result=result)) + failed = True + + if not _is_readonly(handler): + elapsed_s = round(time.perf_counter() - started, 2) + result_dict = result if isinstance(result, dict) else {"value": result} + command = {"action": name, **input_dict} + record: StepRecord | None = None + try: + captured = self.get_env_state( + command=command, + result=result_dict, + elapsed_s=elapsed_s, + ) + except Exception as e: + captured = result_dict + captured["state_capture_error"] = str(e) + captured.setdefault( + "error", f"failed to capture state after {name}: {e}" + ) + captured.setdefault("traceback", traceback.format_exc()) + else: + record = self._state.latest_record() + result = captured + if failed: + for key, value in result_dict.items(): + result.setdefault(key, value) + if record is not None: + self._publish_step(record) + return ToolResult(name=name, result=result) finally: with self._operation_lock: self._active_operation = None operation.done_event.set() + def _publish_step(self, record: StepRecord) -> None: + """Publish one recorded environment step to the dashboard sink.""" + self._dashboard_events.emit( + StepRecordEvent( + record=record, + env_state=self._state, + frame_artifacts=dict(getattr(type(self), "_FRAME_ARTIFACTS", {})), + ) + ) + + def get_env_state( + self, + *, + command: dict[str, Any], + result: dict[str, Any], + elapsed_s: float, + ) -> dict[str, Any]: + """Capture and return the observation produced by a stateful tool.""" + raise NotImplementedError + # ------------------------------------------------------------------ # Server lifecycle hooks (overridden by env toolkits) # ------------------------------------------------------------------ diff --git a/rpent/utils/logging.py b/rpent/utils/logging.py index a0a64f32..94a7c48d 100644 --- a/rpent/utils/logging.py +++ b/rpent/utils/logging.py @@ -87,6 +87,12 @@ def init_output_dir(log_dir: str | Path | None = None, verbose: bool = False) -> if log_dir is None: log_dir = get_repo_root() / "logs" _output_dir = Path(log_dir) + if not _log_initialized and _output_dir.exists() and any(_output_dir.iterdir()): + print( + f"Warning: RPent output directory is not empty: {_output_dir}; existing files may be overwritten!", + file=sys.stderr, + flush=True, + ) _output_dir.mkdir(parents=True, exist_ok=True) if _log_initialized: diff --git a/rpent/utils/sam3_client.py b/rpent/utils/sam3_client.py index f8a80037..22c23977 100644 --- a/rpent/utils/sam3_client.py +++ b/rpent/utils/sam3_client.py @@ -5,7 +5,6 @@ import base64 import io from dataclasses import dataclass -from pathlib import Path from typing import Any import imageio.v2 as imageio @@ -35,7 +34,7 @@ def __init__(self, client: RpcClient, *, timeout_s: float = 120.0) -> None: def segment( self, - image_path: str | Path, + image: bytes | bytearray | memoryview | np.ndarray, *, text_prompt: str | None = None, point: list[int] | None = None, @@ -55,7 +54,12 @@ def segment( if not 0.0 <= float(min_score) <= 1.0: raise ValueError("min_score must be between 0 and 1") - image_bytes = Path(image_path).read_bytes() + if isinstance(image, np.ndarray): + buffer = io.BytesIO() + imageio.imwrite(buffer, image, format="png") + image_bytes = buffer.getvalue() + else: + image_bytes = bytes(image) body: dict[str, Any] = { "image_base64": base64.b64encode(image_bytes).decode("ascii"), "min_score": float(min_score), diff --git a/scripts/codex_proxy/litellm_callbacks.py b/scripts/codex_proxy/litellm_callbacks.py index 23aca5f2..5d54e05b 100644 --- a/scripts/codex_proxy/litellm_callbacks.py +++ b/scripts/codex_proxy/litellm_callbacks.py @@ -92,7 +92,7 @@ def _extract_mcp_namespace_map(data: dict) -> dict[str, str]: Example:: {"back_project": "mcp__rpent", - "view_driver_state": "mcp__rpent"} + "view_env_state": "mcp__rpent"} """ tools = data.get("tools") if not isinstance(tools, list):