Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 2 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -139,11 +139,10 @@ rpent --env libero --suite libero_object_swap --task 2 --seed 0 \

### Live Dashboard

Add `--dashboard` to start a local dashboard server. The command prints the URL in the terminal; open it to confirm the configuration on the launcher screen. Once the run starts, the page streams agent reasoning, camera and Pi0 views, the action timeline, and clip replays. Use `--dashboard-language zh-cn` for the Chinese UI.
Add `--dashboard` to start a local Dashboard and print its URL in the terminal. Open the URL and confirm the configuration; once the services are ready, start a task with `/rpent-task <suite> <task> <seed>`. The page streams agent reasoning, camera views, and the action timeline, and you can submit another task after the current one finishes. Use `--dashboard-language zh-cn` for the Chinese UI.

```bash
rpent --env libero --dashboard --dashboard-language zh-cn \
--suite libero_goal_task --task 1 --seed 0 \
--planner claude_code --model claude-opus-4-8
```

Expand Down Expand Up @@ -172,7 +171,7 @@ For more detailed documentation, see the [RPent documentation](https://rpent.rea
<tr><td><code>--max-episode-steps</code></td><td><code>10000</code></td><td>Max env steps</td></tr>
<tr><td><code>--libero-type</code></td><td><code>LIBERO_TYPE</code> or <code>pro</code></td><td>LIBERO variant: <code>standard</code> | <code>pro</code> | <code>plus</code></td></tr>
<tr><td><code>--cuda-device</code></td><td>inherited</td><td>GPU device exposed to the env / VLA / SAM3 servers</td></tr>
<tr><td><code>--dashboard</code></td><td>off</td><td>Start the local dashboard for this run</td></tr>
<tr><td><code>--dashboard</code></td><td>off</td><td>Start a local Dashboard</td></tr>
<tr><td><code>--dashboard-language</code></td><td><code>en</code></td><td>Dashboard UI language: <code>en</code> | <code>zh-cn</code></td></tr>
<tr><td><code>--env-endpoint</code></td><td>— (spawn)</td><td><code>[protocol://]host:port</code> of an existing env_server (<code>protocol=http|socket</code>, default <code>http</code>). If unset, one is spawned locally.</td></tr>
<tr><td><code>--vla-endpoint</code></td><td>— (spawn)</td><td><code>[protocol://]host:port</code> of an existing vla_server (same rules). If unset, one is spawned locally.</td></tr>
Expand Down
5 changes: 2 additions & 3 deletions README.zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,11 +140,10 @@ rpent --env libero --suite libero_object_swap --task 2 --seed 0 \

### 实时 Dashboard

加上 `--dashboard` 后,会启动本地监控服务,并在终端输出访问地址。打开该地址后,可以在启动页面确认配置;运行开始后,页面会实时显示智能体的推理过程、相机画面和动作时间线。使用 `--dashboard-language zh-cn` 可切换到中文界面。
加上 `--dashboard` 后,会启动本地 Dashboard,并在终端输出访问地址。打开该地址并确认配置;服务就绪后,通过 `/rpent-task <suite> <task> <seed>` 启动任务。页面会实时显示智能体的推理过程、相机画面和动作时间线,任务结束后可以继续提交下一任务。使用 `--dashboard-language zh-cn` 可切换到中文界面。

```bash
rpent --env libero --dashboard --dashboard-language zh-cn \
--suite libero_goal_task --task 1 --seed 0 \
--planner claude_code --model claude-opus-4-8
```

Expand Down Expand Up @@ -184,7 +183,7 @@ bash scripts/run_robocasa.sh PickPlaceCounterToCabinet 0 0 # <任务> <GPU> <
<tr><td><code>--max-episode-steps</code></td><td><code>10000</code></td><td>环境最大步数</td></tr>
<tr><td><code>--libero-type</code></td><td><code>LIBERO_TYPE</code> 或 <code>pro</code></td><td>LIBERO 类型:<code>standard</code> | <code>pro</code> | <code>plus</code></td></tr>
<tr><td><code>--cuda-device</code></td><td>继承当前环境</td><td>env_server、vla_server 和 sam3_server 可见的 GPU 设备</td></tr>
<tr><td><code>--dashboard</code></td><td>关</td><td>为本次运行启动本地 Dashboard</td></tr>
<tr><td><code>--dashboard</code></td><td>关</td><td>启动本地 Dashboard</td></tr>
<tr><td><code>--dashboard-language</code></td><td><code>en</code></td><td>Dashboard 界面语言:<code>en</code> | <code>zh-cn</code></td></tr>
<tr><td><code>--env-endpoint</code></td><td>—(自动启动)</td><td>已在运行的 env_server 的 <code>[protocol://]host:port</code>(<code>protocol=http|socket</code>,默认 <code>http</code>)。留空时自动启动本地实例。</td></tr>
<tr><td><code>--vla-endpoint</code></td><td>—(自动启动)</td><td>已在运行的 vla_server 的 <code>[protocol://]host:port</code>(同上)。留空时自动启动本地实例。</td></tr>
Expand Down
3 changes: 2 additions & 1 deletion docs/source-en/rst_source/development/add_primitive.rst
Original file line number Diff line number Diff line change
Expand Up @@ -132,10 +132,11 @@ primitive requires a few additional components:

.. code-block:: python

def get_toolkit(*, primitives_kwargs, video_path=None):
def get_toolkit(*, primitives_kwargs, dashboard_events, video_path=None):
from robots.myrobot.toolkit import MyRobotToolkit
return MyRobotToolkit(
primitives_kwargs=primitives_kwargs,
dashboard_events=dashboard_events,
video_path=video_path,
)

Expand Down
96 changes: 61 additions & 35 deletions docs/source-en/rst_source/development/add_robot.rst
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,9 @@ order:
4. :ref:`Implement the toolkit and primitives <add-robot-toolkit>`.
5. :ref:`Register environment arguments and build RunConfig
<add-robot-config>`.
6. In :ref:`_init_runtime <add-robot-runtime>`, start or connect to
``env_server`` and any required supporting services.
6. Implement the :ref:`runtime hooks <add-robot-runtime>`: one complete
runtime for normal CLI runs, plus the Dashboard-only Session/TaskRun split
if the environment supports Dashboard task control.

.. _add-robot-entry:

Expand All @@ -49,6 +50,7 @@ two factory functions:
.. code-block:: python

# robots/myenv/__init__.py
from rpent.dashboard.events import DashboardEventSink
from rpent.envs.env_spec import EnvSpec, RunConfig
from rpent.envs.prompt_bundle import PromptBundle
from robots.myenv.prompt_bundle import system_prompt, user_prompt
Expand All @@ -59,12 +61,18 @@ two factory functions:
prompts=PromptBundle(system=system_prompt, user=user_prompt),
add_cli_args=_add_cli_args,
parse_config=_parse_config,
init_shared_runtime=_init_shared_runtime,
init_task_runtime=_init_task_runtime,
init_runtime=_init_runtime,
)

def get_toolkit(*, primitives_kwargs, video_path=None):
def get_toolkit(*, primitives_kwargs, dashboard_events: DashboardEventSink, video_path=None):
from robots.myenv.toolkit import MyEnvToolkit
return MyEnvToolkit(primitives_kwargs=primitives_kwargs, video_path=video_path)
return MyEnvToolkit(
primitives_kwargs=primitives_kwargs,
dashboard_events=dashboard_events,
video_path=video_path,
)

def _add_cli_args(parser, use_dashboard) -> None:
"""Register env flags on the shared parser. See §4."""
Expand All @@ -74,20 +82,28 @@ two factory functions:
"""Validate final `args`, return a RunConfig. See §4."""
...

def _init_runtime(args, output_dir):
"""Spawn env_server, vla_server, and any supporting services.
def _init_runtime(args, output_dir, dashboard_events: DashboardEventSink):
"""Normal CLI only: initialize the complete runtime.

Returns (daemons, primitives_kwargs). See §5.
"""
...

def _init_shared_runtime(args, output_dir, dashboard_events: DashboardEventSink):
"""Dashboard only: initialize Session-owned reusable services."""
...

def _init_task_runtime(args, output_dir, dashboard_events: DashboardEventSink):
"""Dashboard only: initialize fresh per-TaskRun services."""
...

That's the entire registration step — ``_resolve_env(name)`` does an
``importlib.import_module(f"robots.{name}")``, so dropping the package under
``robots/`` on disk is enough. No central list to update.

The sections below describe what each referenced module must contain.
``_add_cli_args`` / ``_parse_config`` are covered in §4 and ``_init_runtime``
in §5.
``_add_cli_args`` / ``_parse_config`` are covered in §4 and the three runtime
hooks in §5.

.. _add-robot-env-rpc:

Expand Down Expand Up @@ -268,13 +284,15 @@ hooks and participate in the final argparse pass:
**``_add_cli_args(parser, use_dashboard) -> None``.** Register the
environment's arguments on the shared parser created by main.py.
``use_dashboard`` determines whether normally required arguments remain
optional for the dashboard launcher to fill in later. main.py calls this hook
optional. For each Dashboard TaskRun, the ``/rpent-task`` command supplies
``suite`` and ``task`` before ``parse_config`` runs. main.py calls this hook
before ``parser.parse_args()``, so argparse's usage and error output includes
the environment arguments.

**``_parse_config(args) -> RunConfig``.** Called after ``parser.parse_args()``
and, if applicable, the dashboard launcher. Enforces any dashboard-only
optional flags are now populated and returns a
**``_parse_config(args) -> RunConfig``.** In normal CLI mode, this is called
after ``parser.parse_args()``. In Dashboard mode, it is called for each
TaskRun after the ``/rpent-task`` command's ``suite`` and ``task`` have been
copied to the task arguments. It validates those fields and returns a
:class:`~rpent.envs.RunConfig`:

- ``recipe_tag`` — env's per-run tag, used in transcript filenames / recipe
Expand All @@ -283,8 +301,6 @@ optional flags are now populated and returns a
calls ``init_output_dir`` to create it and configure logging).
- ``prompt_vars`` — dict passed to ``PromptBundle.render`` (typically the run
identifiers plus anything else the prompts reference).
- ``dashboard_state`` — a :class:`~rpent.dashboard.state.State` when
``args.dashboard`` is set, else ``None``.
- ``task_desc`` — env-specific dict of task-identifying fields, written into
the transcript JSON record verbatim (LIBERO:
``{"suite": ..., "task": ..., "seed": ...}``).
Expand All @@ -299,38 +315,48 @@ optional flags are now populated and returns a

def _parse_config(args) -> RunConfig:
if not args.suite: raise ValueError("--suite is required")
# ... derive recipe_tag, output_dir, prompt_vars, dashboard_state ...
# ... derive recipe_tag, output_dir, and prompt_vars ...
return RunConfig(
recipe_tag=recipe_tag,
output_dir=output_dir,
prompt_vars=prompt_vars,
dashboard_state=dashboard_state,
task_desc={"suite": args.suite, "task": args.task, "seed": args.seed},
)

.. _add-robot-runtime:

5. ``_init_runtime`` (runner hook)
----------------------------------

After ``parse_config`` returns, main.py calls
``env_spec.init_runtime(args, output_dir)`` to initialize the environment and
VLA services and build the toolkit inputs. The environment can spawn as many
subprocesses as it needs. The current LIBERO implementation starts
``env_server``, ``vla_server``, and ``sam3_server``. The hook returns
``(daemons, primitives_kwargs)``:

- ``daemons: list[ProcessDaemon]`` — subprocesses owned by this run; main.py
calls ``.stop()`` on each one in its ``finally`` block.
- ``primitives_kwargs: dict`` — passed verbatim to the toolkit constructor
(which forwards it to the primitives' ``__init__``). It typically
contains ``{"env": MyEnvClient(...), "model": VLAClient(...)}``; add clients
for any supporting services here as well, such as LIBERO's ``sam3_client``.
5. Runtime initialization hooks
-------------------------------

All runtime hooks return ``(owned_daemons, primitives_kwargs)``:

- ``owned_daemons: list[ProcessDaemon]`` contains only subprocesses started
by this process. The active runner stops them during cleanup. A client for an
external endpoint must not add that external service to this list.
- ``primitives_kwargs: dict`` is passed to the toolkit constructor, which
forwards it to the primitives' ``__init__``. A complete set commonly
contains ``{"env": MyEnvClient(...), "model": VLAClient(...)}`` plus any
supporting clients.

``init_runtime`` is the normal CLI hook. After ``parse_config`` returns,
``main.py`` calls it once to initialize the complete runtime. The current
LIBERO implementation starts or attaches to ``env_server``, ``vla_server``,
and ``sam3_server`` and returns all primitive inputs together.

``init_shared_runtime`` and ``init_task_runtime`` are **Dashboard-only**
hooks; the normal CLI path never calls them. The Dashboard calls
``init_shared_runtime`` once for Session-owned services, then calls
``init_task_runtime`` for every fresh TaskRun and merges the two returned
``primitives_kwargs`` dictionaries. The split is environment-specific. For
LIBERO, VLA and SAM3 are Session-owned while the environment is TaskRun-owned;
another environment should use the lifecycle split appropriate to its own

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR #69 still hardcodes /rpent-task to LIBERO suites and suite/task/seed, so implementing these hooks is not enough for another environment to support dashboard task control. Could we document this flow as LIBERO-only for now?

services rather than copying that arrangement mechanically.

Endpoint parsing (``--env-endpoint``, ``--vla-endpoint``, and LIBERO's
``--sam3-endpoint``) and subprocess spawning (``--cuda-device`` passthrough,
``MUJOCO_GL``, ...) live here — main.py knows nothing about them. See
``robots/libero/__init__.py`` for the reference implementation.
``--sam3-endpoint``), subprocess spawning, and runtime status events belong in
the hook that owns the corresponding service. The runners do not handle those
environment details. See ``robots/libero/__init__.py`` for the reference
implementation.

Smoke test
----------
Expand Down
71 changes: 39 additions & 32 deletions docs/source-en/rst_source/development/architecture.rst
Original file line number Diff line number Diff line change
Expand Up @@ -103,31 +103,31 @@ components required for a run. On startup, it:
2. Resolves the env via ``get_env_spec(args.env_name)`` and calls
``env_spec.add_cli_args(parser, use_dashboard=args.dashboard)`` — the env
registers its flags on the shared parser. ``use_dashboard=True`` makes
its otherwise-required flags optional so the dashboard can supply them.
task-specific flags optional because the Dashboard receives them later
through task commands.
3. Runs ``parser.parse_args()`` against the complete parser to perform
argparse-level validation and produce the final ``args``, retaining
argparse's standard usage and error output.
4. If ``--dashboard`` is set, starts the launcher with the current arguments
as defaults and applies the submitted configuration back to ``args``.
5. Calls ``env_spec.parse_config(args)`` to validate the run configuration
and produce a
:class:`~rpent.envs.RunConfig`
(``recipe_tag`` / ``output_dir`` / ``prompt_vars`` / ``dashboard_state``
/ ``task_desc``). Under ``--dashboard``, this is where the env
enforces that its previously-optional flags were actually filled in.
4. If ``--dashboard`` is set, hands control to ``rpent/cli/dashboard.py`` and
returns when that long-lived Session ends. The Dashboard-only lifecycle is
described below; the remaining steps are the normal CLI path.
5. Calls ``env_spec.parse_config(args)`` to validate the normal CLI run
configuration
and produce a :class:`~rpent.envs.RunConfig` (``recipe_tag`` /
``output_dir`` / ``prompt_vars`` / ``task_desc``).
6. Calls ``init_output_dir`` to create the run's output directory and
configure ``run.log``.
7. Builds the **planner** through ``rpent.planner.base.build_planner`` based
on ``--planner``, then renders the system and user prompts from the env's
prompt bundle.
8. Calls ``env_spec.init_runtime(args, output_dir)``. The env implementation
starts ``env_server`` and ``vla_server``, or connects to existing services
when ``--env-endpoint`` / ``--vla-endpoint`` is supplied, and returns
``(daemons, primitives_kwargs)``.
9. Passes ``primitives_kwargs`` to the env's ``get_toolkit`` factory to
construct the **toolkit**.
10. Runs the tool-calling loop, streams to the dashboard if
``--dashboard`` is set, and then writes
8. Calls ``env_spec.init_runtime(args, output_dir, dashboard_events)``. The env
implementation starts ``env_server``, ``vla_server``, and ``sam3_server``, or connects
to existing services when the corresponding endpoint is supplied, and
returns ``(daemons, primitives_kwargs)``.
9. Passes ``primitives_kwargs`` and a ``dashboard_events`` sink to the env's
``get_toolkit`` factory to construct the **toolkit**. The one-shot path
uses a no-op event sink.
10. Runs the tool-calling loop, then writes
``<output_dir>/transcript_*.json`` and flushes toolkit recordings during
cleanup.

Expand All @@ -149,11 +149,12 @@ two factories exposed by that package:
# robots/myenv/__init__.py
def get_env_spec() -> EnvSpec: ... # identity, prompt bundle, and runner hooks
def get_toolkit(
*, primitives_kwargs, video_path=None, dashboard=None
*, primitives_kwargs, dashboard_events, video_path=None
): ...

``EnvSpec`` gathers the environment's identity, its prompt templates, and the
three runner hooks (``add_cli_args`` / ``parse_config`` / ``init_runtime``); see
``EnvSpec`` gathers the environment's identity, its prompt templates, and five
runner hooks: ``add_cli_args`` / ``parse_config`` / ``init_runtime``, plus the
Dashboard-only ``init_shared_runtime`` / ``init_task_runtime`` pair. See
:doc:`interfaces` for what each field must provide.

The loader itself does not maintain a list of environment names. The
Expand All @@ -180,23 +181,29 @@ Dashboard (optional)
--------------------

``rpent/dashboard/`` contains a FastAPI application and a static
frontend. With ``--dashboard``, ``rpent/cli/main.py`` starts the
Dashboard using ``--dashboard-host`` and ``--dashboard-port``. It binds
to ``127.0.0.1`` by default and lets the operating system choose a free
port. Before the run starts, the launcher lets the user review or change
the configuration.

During the run, the Dashboard shows:
frontend. With ``--dashboard``, ``rpent/cli/main.py`` hands control to
``rpent/cli/dashboard.py``, which starts the Dashboard with
``--dashboard-host`` and ``--dashboard-port`` and confirms the configuration
before calling the Dashboard-only ``env_spec.init_shared_runtime`` hook once.
The Session controller then waits for ``/rpent-task`` commands. For every
claimed TaskRun, the Dashboard calls ``parse_config`` and the Dashboard-only
``env_spec.init_task_runtime`` hook, merges the shared and task primitive
inputs, and creates a fresh toolkit and planner conversation. In LIBERO, VLA
and SAM3 are reused while the Dashboard is running, while every TaskRun gets a
separate environment runtime and executes sequentially.

During a TaskRun, the Dashboard shows:

- planner output and tool-call events;
- live camera and Pi0.5 views;
- live fixed-camera and wrist-camera views;
- the action timeline and per-action clips;
- the complete episode recording after the run, if one was generated.

The server sends state summaries over SSE, and the frontend fetches
detailed events, timeline data, and images as needed. The Dashboard
displays state produced by the planner and toolkit; it does not issue
robot actions directly.
The page accepts ordinary planner messages, new task commands, and interrupt
requests, but these controls do not issue robot actions directly. Planners,
toolkits, and environment runtimes publish display updates through a
``dashboard_events`` sink. The server sends state summaries over SSE, and the
frontend fetches detailed events, timeline data, and images as needed.

Next steps
----------
Expand Down
Loading