diff --git a/README.md b/README.md index 1107daee..83c23b5e 100644 --- a/README.md +++ b/README.md @@ -139,11 +139,10 @@ rpent --env libero --suite libero_object_swap --task 2 --seed 0 \ ### Live Dashboard -Add `--dashboard` to start a local dashboard server. The command prints the URL in the terminal; open it to confirm the configuration on the launcher screen. Once the run starts, the page streams agent reasoning, camera and Pi0 views, the action timeline, and clip replays. Use `--dashboard-language zh-cn` for the Chinese UI. +Add `--dashboard` to start a local Dashboard and print its URL in the terminal. Open the URL and confirm the configuration; once the services are ready, start a task with `/rpent-task `. The page streams agent reasoning, camera views, and the action timeline, and you can submit another task after the current one finishes. Use `--dashboard-language zh-cn` for the Chinese UI. ```bash rpent --env libero --dashboard --dashboard-language zh-cn \ - --suite libero_goal_task --task 1 --seed 0 \ --planner claude_code --model claude-opus-4-8 ``` @@ -172,7 +171,7 @@ For more detailed documentation, see the [RPent documentation](https://rpent.rea --max-episode-steps10000Max env steps --libero-typeLIBERO_TYPE or proLIBERO variant: standard | pro | plus --cuda-deviceinheritedGPU device exposed to the env / VLA / SAM3 servers - --dashboardoffStart the local dashboard for this run + --dashboardoffStart a local Dashboard --dashboard-languageenDashboard UI language: en | zh-cn --env-endpoint— (spawn)[protocol://]host:port of an existing env_server (protocol=http|socket, default http). If unset, one is spawned locally. --vla-endpoint— (spawn)[protocol://]host:port of an existing vla_server (same rules). If unset, one is spawned locally. diff --git a/README.zh-CN.md b/README.zh-CN.md index a8e30efa..e9cbe0d0 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -140,11 +140,10 @@ rpent --env libero --suite libero_object_swap --task 2 --seed 0 \ ### 实时 Dashboard -加上 `--dashboard` 后，会启动本地监控服务，并在终端输出访问地址。打开该地址后，可以在启动页面确认配置；运行开始后，页面会实时显示智能体的推理过程、相机画面和动作时间线。使用 `--dashboard-language zh-cn` 可切换到中文界面。 +加上 `--dashboard` 后，会启动本地 Dashboard，并在终端输出访问地址。打开该地址并确认配置；服务就绪后，通过 `/rpent-task ` 启动任务。页面会实时显示智能体的推理过程、相机画面和动作时间线，任务结束后可以继续提交下一任务。使用 `--dashboard-language zh-cn` 可切换到中文界面。 ```bash rpent --env libero --dashboard --dashboard-language zh-cn \ - --suite libero_goal_task --task 1 --seed 0 \ --planner claude_code --model claude-opus-4-8 ``` @@ -184,7 +183,7 @@ bash scripts/run_robocasa.sh PickPlaceCounterToCabinet 0 0 # <任务> < --max-episode-steps10000环境最大步数 --libero-typeLIBERO_TYPE 或 proLIBERO 类型：standard | pro | plus --cuda-device继承当前环境env_server、vla_server 和 sam3_server 可见的 GPU 设备 - --dashboard关为本次运行启动本地 Dashboard + --dashboard关启动本地 Dashboard --dashboard-languageenDashboard 界面语言：en | zh-cn --env-endpoint—（自动启动）已在运行的 env_server 的 [protocol://]host:port（protocol=http|socket，默认 http）。留空时自动启动本地实例。 --vla-endpoint—（自动启动）已在运行的 vla_server 的 [protocol://]host:port（同上）。留空时自动启动本地实例。 diff --git a/docs/source-en/rst_source/development/add_primitive.rst b/docs/source-en/rst_source/development/add_primitive.rst index dcefdd5b..030d6c0d 100644 --- a/docs/source-en/rst_source/development/add_primitive.rst +++ b/docs/source-en/rst_source/development/add_primitive.rst @@ -132,10 +132,11 @@ primitive requires a few additional components: .. code-block:: python - def get_toolkit(*, primitives_kwargs, video_path=None): + def get_toolkit(*, primitives_kwargs, dashboard_events, video_path=None): from robots.myrobot.toolkit import MyRobotToolkit return MyRobotToolkit( primitives_kwargs=primitives_kwargs, + dashboard_events=dashboard_events, video_path=video_path, ) diff --git a/docs/source-en/rst_source/development/add_robot.rst b/docs/source-en/rst_source/development/add_robot.rst index 2111a9f7..b5836e97 100644 --- a/docs/source-en/rst_source/development/add_robot.rst +++ b/docs/source-en/rst_source/development/add_robot.rst @@ -22,8 +22,9 @@ order: 4. :ref:`Implement the toolkit and primitives `. 5. :ref:`Register environment arguments and build RunConfig `. -6. In :ref:`_init_runtime `, start or connect to - ``env_server`` and any required supporting services. +6. Implement the :ref:`runtime hooks `: one complete + runtime for normal CLI runs, plus the Dashboard-only Session/TaskRun split + if the environment supports Dashboard task control. .. _add-robot-entry: @@ -49,6 +50,7 @@ two factory functions: .. code-block:: python # robots/myenv/__init__.py + from rpent.dashboard.events import DashboardEventSink from rpent.envs.env_spec import EnvSpec, RunConfig from rpent.envs.prompt_bundle import PromptBundle from robots.myenv.prompt_bundle import system_prompt, user_prompt @@ -59,12 +61,18 @@ two factory functions: prompts=PromptBundle(system=system_prompt, user=user_prompt), add_cli_args=_add_cli_args, parse_config=_parse_config, + init_shared_runtime=_init_shared_runtime, + init_task_runtime=_init_task_runtime, init_runtime=_init_runtime, ) - def get_toolkit(*, primitives_kwargs, video_path=None): + def get_toolkit(*, primitives_kwargs, dashboard_events: DashboardEventSink, video_path=None): from robots.myenv.toolkit import MyEnvToolkit - return MyEnvToolkit(primitives_kwargs=primitives_kwargs, video_path=video_path) + return MyEnvToolkit( + primitives_kwargs=primitives_kwargs, + dashboard_events=dashboard_events, + video_path=video_path, + ) def _add_cli_args(parser, use_dashboard) -> None: """Register env flags on the shared parser. See §4.""" @@ -74,20 +82,28 @@ two factory functions: """Validate final `args`, return a RunConfig. See §4.""" ... - def _init_runtime(args, output_dir): - """Spawn env_server, vla_server, and any supporting services. + def _init_runtime(args, output_dir, dashboard_events: DashboardEventSink): + """Normal CLI only: initialize the complete runtime. Returns (daemons, primitives_kwargs). See §5. """ ... + def _init_shared_runtime(args, output_dir, dashboard_events: DashboardEventSink): + """Dashboard only: initialize Session-owned reusable services.""" + ... + + def _init_task_runtime(args, output_dir, dashboard_events: DashboardEventSink): + """Dashboard only: initialize fresh per-TaskRun services.""" + ... + That's the entire registration step — ``_resolve_env(name)`` does an ``importlib.import_module(f"robots.{name}")``, so dropping the package under ``robots/`` on disk is enough. No central list to update. The sections below describe what each referenced module must contain. -``_add_cli_args`` / ``_parse_config`` are covered in §4 and ``_init_runtime`` -in §5. +``_add_cli_args`` / ``_parse_config`` are covered in §4 and the three runtime +hooks in §5. .. _add-robot-env-rpc: @@ -268,13 +284,15 @@ hooks and participate in the final argparse pass: **``_add_cli_args(parser, use_dashboard) -> None``.** Register the environment's arguments on the shared parser created by main.py. ``use_dashboard`` determines whether normally required arguments remain -optional for the dashboard launcher to fill in later. main.py calls this hook +optional. For each Dashboard TaskRun, the ``/rpent-task`` command supplies +``suite`` and ``task`` before ``parse_config`` runs. main.py calls this hook before ``parser.parse_args()``, so argparse's usage and error output includes the environment arguments. -**``_parse_config(args) -> RunConfig``.** Called after ``parser.parse_args()`` -and, if applicable, the dashboard launcher. Enforces any dashboard-only -optional flags are now populated and returns a +**``_parse_config(args) -> RunConfig``.** In normal CLI mode, this is called +after ``parser.parse_args()``. In Dashboard mode, it is called for each +TaskRun after the ``/rpent-task`` command's ``suite`` and ``task`` have been +copied to the task arguments. It validates those fields and returns a :class:`~rpent.envs.RunConfig`: - ``recipe_tag`` — env's per-run tag, used in transcript filenames / recipe @@ -283,8 +301,6 @@ optional flags are now populated and returns a calls ``init_output_dir`` to create it and configure logging). - ``prompt_vars`` — dict passed to ``PromptBundle.render`` (typically the run identifiers plus anything else the prompts reference). -- ``dashboard_state`` — a :class:`~rpent.dashboard.state.State` when - ``args.dashboard`` is set, else ``None``. - ``task_desc`` — env-specific dict of task-identifying fields, written into the transcript JSON record verbatim (LIBERO: ``{"suite": ..., "task": ..., "seed": ...}``). @@ -299,38 +315,48 @@ optional flags are now populated and returns a def _parse_config(args) -> RunConfig: if not args.suite: raise ValueError("--suite is required") - # ... derive recipe_tag, output_dir, prompt_vars, dashboard_state ... + # ... derive recipe_tag, output_dir, and prompt_vars ... return RunConfig( recipe_tag=recipe_tag, output_dir=output_dir, prompt_vars=prompt_vars, - dashboard_state=dashboard_state, task_desc={"suite": args.suite, "task": args.task, "seed": args.seed}, ) .. _add-robot-runtime: -5. ``_init_runtime`` (runner hook) ----------------------------------- - -After ``parse_config`` returns, main.py calls -``env_spec.init_runtime(args, output_dir)`` to initialize the environment and -VLA services and build the toolkit inputs. The environment can spawn as many -subprocesses as it needs. The current LIBERO implementation starts -``env_server``, ``vla_server``, and ``sam3_server``. The hook returns -``(daemons, primitives_kwargs)``: - -- ``daemons: list[ProcessDaemon]`` — subprocesses owned by this run; main.py - calls ``.stop()`` on each one in its ``finally`` block. -- ``primitives_kwargs: dict`` — passed verbatim to the toolkit constructor - (which forwards it to the primitives' ``__init__``). It typically - contains ``{"env": MyEnvClient(...), "model": VLAClient(...)}``; add clients - for any supporting services here as well, such as LIBERO's ``sam3_client``. +5. Runtime initialization hooks +------------------------------- + +All runtime hooks return ``(owned_daemons, primitives_kwargs)``: + +- ``owned_daemons: list[ProcessDaemon]`` contains only subprocesses started + by this process. The active runner stops them during cleanup. A client for an + external endpoint must not add that external service to this list. +- ``primitives_kwargs: dict`` is passed to the toolkit constructor, which + forwards it to the primitives' ``__init__``. A complete set commonly + contains ``{"env": MyEnvClient(...), "model": VLAClient(...)}`` plus any + supporting clients. + +``init_runtime`` is the normal CLI hook. After ``parse_config`` returns, +``main.py`` calls it once to initialize the complete runtime. The current +LIBERO implementation starts or attaches to ``env_server``, ``vla_server``, +and ``sam3_server`` and returns all primitive inputs together. + +``init_shared_runtime`` and ``init_task_runtime`` are **Dashboard-only** +hooks; the normal CLI path never calls them. The Dashboard calls +``init_shared_runtime`` once for Session-owned services, then calls +``init_task_runtime`` for every fresh TaskRun and merges the two returned +``primitives_kwargs`` dictionaries. The split is environment-specific. For +LIBERO, VLA and SAM3 are Session-owned while the environment is TaskRun-owned; +another environment should use the lifecycle split appropriate to its own +services rather than copying that arrangement mechanically. Endpoint parsing (``--env-endpoint``, ``--vla-endpoint``, and LIBERO's -``--sam3-endpoint``) and subprocess spawning (``--cuda-device`` passthrough, -``MUJOCO_GL``, ...) live here — main.py knows nothing about them. See -``robots/libero/__init__.py`` for the reference implementation. +``--sam3-endpoint``), subprocess spawning, and runtime status events belong in +the hook that owns the corresponding service. The runners do not handle those +environment details. See ``robots/libero/__init__.py`` for the reference +implementation. Smoke test ---------- diff --git a/docs/source-en/rst_source/development/architecture.rst b/docs/source-en/rst_source/development/architecture.rst index c00b0467..87458dad 100644 --- a/docs/source-en/rst_source/development/architecture.rst +++ b/docs/source-en/rst_source/development/architecture.rst @@ -103,31 +103,31 @@ components required for a run. On startup, it: 2. Resolves the env via ``get_env_spec(args.env_name)`` and calls ``env_spec.add_cli_args(parser, use_dashboard=args.dashboard)`` — the env registers its flags on the shared parser. ``use_dashboard=True`` makes - its otherwise-required flags optional so the dashboard can supply them. + task-specific flags optional because the Dashboard receives them later + through task commands. 3. Runs ``parser.parse_args()`` against the complete parser to perform argparse-level validation and produce the final ``args``, retaining argparse's standard usage and error output. -4. If ``--dashboard`` is set, starts the launcher with the current arguments - as defaults and applies the submitted configuration back to ``args``. -5. Calls ``env_spec.parse_config(args)`` to validate the run configuration - and produce a - :class:`~rpent.envs.RunConfig` - (``recipe_tag`` / ``output_dir`` / ``prompt_vars`` / ``dashboard_state`` - / ``task_desc``). Under ``--dashboard``, this is where the env - enforces that its previously-optional flags were actually filled in. +4. If ``--dashboard`` is set, hands control to ``rpent/cli/dashboard.py`` and + returns when that long-lived Session ends. The Dashboard-only lifecycle is + described below; the remaining steps are the normal CLI path. +5. Calls ``env_spec.parse_config(args)`` to validate the normal CLI run + configuration + and produce a :class:`~rpent.envs.RunConfig` (``recipe_tag`` / + ``output_dir`` / ``prompt_vars`` / ``task_desc``). 6. Calls ``init_output_dir`` to create the run's output directory and configure ``run.log``. 7. Builds the **planner** through ``rpent.planner.base.build_planner`` based on ``--planner``, then renders the system and user prompts from the env's prompt bundle. -8. Calls ``env_spec.init_runtime(args, output_dir)``. The env implementation - starts ``env_server`` and ``vla_server``, or connects to existing services - when ``--env-endpoint`` / ``--vla-endpoint`` is supplied, and returns - ``(daemons, primitives_kwargs)``. -9. Passes ``primitives_kwargs`` to the env's ``get_toolkit`` factory to - construct the **toolkit**. -10. Runs the tool-calling loop, streams to the dashboard if - ``--dashboard`` is set, and then writes +8. Calls ``env_spec.init_runtime(args, output_dir, dashboard_events)``. The env + implementation starts ``env_server``, ``vla_server``, and ``sam3_server``, or connects + to existing services when the corresponding endpoint is supplied, and + returns ``(daemons, primitives_kwargs)``. +9. Passes ``primitives_kwargs`` and a ``dashboard_events`` sink to the env's + ``get_toolkit`` factory to construct the **toolkit**. The one-shot path + uses a no-op event sink. +10. Runs the tool-calling loop, then writes ``/transcript_*.json`` and flushes toolkit recordings during cleanup. @@ -149,11 +149,12 @@ two factories exposed by that package: # robots/myenv/__init__.py def get_env_spec() -> EnvSpec: ... # identity, prompt bundle, and runner hooks def get_toolkit( - *, primitives_kwargs, video_path=None, dashboard=None + *, primitives_kwargs, dashboard_events, video_path=None ): ... -``EnvSpec`` gathers the environment's identity, its prompt templates, and the -three runner hooks (``add_cli_args`` / ``parse_config`` / ``init_runtime``); see +``EnvSpec`` gathers the environment's identity, its prompt templates, and five +runner hooks: ``add_cli_args`` / ``parse_config`` / ``init_runtime``, plus the +Dashboard-only ``init_shared_runtime`` / ``init_task_runtime`` pair. See :doc:`interfaces` for what each field must provide. The loader itself does not maintain a list of environment names. The @@ -180,23 +181,29 @@ Dashboard (optional) -------------------- ``rpent/dashboard/`` contains a FastAPI application and a static -frontend. With ``--dashboard``, ``rpent/cli/main.py`` starts the -Dashboard using ``--dashboard-host`` and ``--dashboard-port``. It binds -to ``127.0.0.1`` by default and lets the operating system choose a free -port. Before the run starts, the launcher lets the user review or change -the configuration. - -During the run, the Dashboard shows: +frontend. With ``--dashboard``, ``rpent/cli/main.py`` hands control to +``rpent/cli/dashboard.py``, which starts the Dashboard with +``--dashboard-host`` and ``--dashboard-port`` and confirms the configuration +before calling the Dashboard-only ``env_spec.init_shared_runtime`` hook once. +The Session controller then waits for ``/rpent-task`` commands. For every +claimed TaskRun, the Dashboard calls ``parse_config`` and the Dashboard-only +``env_spec.init_task_runtime`` hook, merges the shared and task primitive +inputs, and creates a fresh toolkit and planner conversation. In LIBERO, VLA +and SAM3 are reused while the Dashboard is running, while every TaskRun gets a +separate environment runtime and executes sequentially. + +During a TaskRun, the Dashboard shows: - planner output and tool-call events; -- live camera and Pi0.5 views; +- live fixed-camera and wrist-camera views; - the action timeline and per-action clips; - the complete episode recording after the run, if one was generated. -The server sends state summaries over SSE, and the frontend fetches -detailed events, timeline data, and images as needed. The Dashboard -displays state produced by the planner and toolkit; it does not issue -robot actions directly. +The page accepts ordinary planner messages, new task commands, and interrupt +requests, but these controls do not issue robot actions directly. Planners, +toolkits, and environment runtimes publish display updates through a +``dashboard_events`` sink. The server sends state summaries over SSE, and the +frontend fetches detailed events, timeline data, and images as needed. Next steps ---------- diff --git a/docs/source-en/rst_source/development/interfaces.rst b/docs/source-en/rst_source/development/interfaces.rst index bf6a6a59..628d9609 100644 --- a/docs/source-en/rst_source/development/interfaces.rst +++ b/docs/source-en/rst_source/development/interfaces.rst @@ -12,7 +12,7 @@ After you add ``robots//``, ``main.py`` calls two functions in ``__init__.p .. code-block:: python def get_env_spec() -> EnvSpec: ... - def get_toolkit(*, primitives_kwargs, video_path=None, dashboard=None): ... + def get_toolkit(*, primitives_kwargs, dashboard_events: DashboardEventSink, video_path=None): ... ``get_env_spec`` returns an ``EnvSpec``. You supply: @@ -33,11 +33,19 @@ After you add ``robots//``, ``main.py`` calls two functions in ``__init__.p - Validate args and return ``RunConfig``; set at least ``recipe_tag``, ``output_dir``, and ``prompt_vars`` for prompt templating. * - ``init_runtime`` - - Start or attach to env / VLA subprocesses; build ``primitives_kwargs`` - (env client, model client, etc.) for the toolkit's primitives. + - Normal CLI only: start or attach to the complete runtime and build + ``primitives_kwargs`` (env client, model client, etc.) for the toolkit's + primitives. A ``DashboardEventSink`` reports runtime status. + * - ``init_shared_runtime`` + - Dashboard only: initialize Session-owned services that can be reused by + multiple TaskRuns, and return their owned daemons and primitive inputs. + * - ``init_task_runtime`` + - Dashboard only: initialize the fresh per-TaskRun services and return + their owned daemons and primitive inputs. ``get_toolkit`` usually just passes ``primitives_kwargs`` into your env subclass; -``video_path`` and ``dashboard`` are passed by ``main.py`` — you rarely touch them. +``dashboard_events`` and ``video_path`` are supplied by the active runner, so +you normally do not need to change them. Reference: ``robots/libero/__init__.py``. @@ -58,6 +66,7 @@ Most users pick a built-in ``api``, ``claude_code``, or ``codex`` planner — se toolkit: Toolkit, max_turns: int, input_queue=None, + dashboard_interaction=None, ) -> PlannerResult: ... Contract: pass ``toolkit.get_tools_spec()`` to the model; dispatch each call via @@ -98,7 +107,8 @@ Inter-process communication Relevant when attaching to existing servers or writing ``env_server`` / ``vla_server``. -Client endpoints — expose in ``add_cli_args`` or parse in ``init_runtime``: +Client endpoints — expose in ``add_cli_args`` and parse in the applicable +normal-CLI or Dashboard runtime hook: .. code-block:: text diff --git a/docs/source-en/rst_source/quickstart.rst b/docs/source-en/rst_source/quickstart.rst index b9ba30b5..a57f8446 100644 --- a/docs/source-en/rst_source/quickstart.rst +++ b/docs/source-en/rst_source/quickstart.rst @@ -43,18 +43,19 @@ To switch to another planner, such as ``codex`` or ``api``, see 3. Monitor the run in the Dashboard ----------------------------------- -Add ``--dashboard`` to start a local Dashboard service and print its URL -in the terminal. Open the URL to confirm the configuration on the -launcher screen. Once the run starts, the page streams the agent's -reasoning, live camera and Pi0 views, an action timeline, and clip -replays. Use ``--dashboard-language zh-cn`` for the Chinese UI. +Add ``--dashboard`` to start a local Dashboard and print its URL in the terminal: .. code-block:: bash rpent --env libero --dashboard --dashboard-language zh-cn \ - --suite libero_object_swap --task 2 --seed 0 \ --planner claude_code --model claude-opus-4-8 +Open the URL and confirm the configuration. Once the services are ready, enter +``/rpent-task libero_object_swap 2 0`` in the page to start a task. The Dashboard +streams agent reasoning, camera views, and the action timeline; submit another +task after the current one finishes. Use ``--dashboard-language zh-cn`` for the +Chinese UI. + Key CLI options --------------- diff --git a/docs/source-en/rst_source/usage/advanced_deployment.rst b/docs/source-en/rst_source/usage/advanced_deployment.rst index 78d699b9..5c265e1b 100644 --- a/docs/source-en/rst_source/usage/advanced_deployment.rst +++ b/docs/source-en/rst_source/usage/advanced_deployment.rst @@ -11,6 +11,10 @@ environment, ``--vla-endpoint`` for the Pi0.5 VLA, and ``--sam3-endpoint`` for SAM3. Each takes ``[protocol://]HOST:PORT`` — HTTP when the protocol is omitted, or ``socket://`` for socket RPC. +Dashboard Sessions do not support ``--env-endpoint`` because every TaskRun +uses a fresh environment service. ``--vla-endpoint`` and ``--sam3-endpoint`` +remain available in Dashboard mode. + LIBERO environment service -------------------------- diff --git a/docs/source-en/rst_source/usage/libero.rst b/docs/source-en/rst_source/usage/libero.rst index 4fd25442..0e25f106 100644 --- a/docs/source-en/rst_source/usage/libero.rst +++ b/docs/source-en/rst_source/usage/libero.rst @@ -156,18 +156,33 @@ These tools do not advance the environment. Live dashboard -------------- -Add ``--dashboard`` to start a local monitor. It selects an available -port and prints the URL in the terminal: +Add ``--dashboard`` to start a long-lived local Dashboard Session. It +selects an available port and prints the URL in the terminal: .. code-block:: bash rpent --env libero --dashboard \ - --suite libero_object_swap --task 2 --seed 0 \ --planner claude_code --model claude-opus-4-8 -The dashboard streams reasoning, agentview + wrist camera + Pi0.5 -overlays, and an action timeline. Use -``--dashboard-language zh-cn`` for the Chinese UI. +Open the URL, confirm the Session configuration, and click **Start Session**. +After the shared services are ready, start a TaskRun from the page with: + +.. code-block:: text + + /rpent-task libero_object_swap 2 0 + +Each TaskRun gets a fresh environment while the VLA and SAM3 services are +reused by the Session. Submit another ``/rpent-task`` after completion, or +submit one while a task is running to switch tasks. The Dashboard streams +agent reasoning and tool calls, fixed and wrist camera views, the action +timeline, and action and full-episode replays. During a TaskRun, normal +messages steer the agent and Esc requests an interruption. Press Ctrl+C in +the terminal to stop the Session. + +``--dashboard`` cannot be combined with ``--interactive`` or +``--env-endpoint``. External ``--vla-endpoint`` and ``--sam3-endpoint`` +services remain supported. Use ``--dashboard-language zh-cn`` for the +Chinese UI. Bringing your own VLA --------------------- diff --git a/docs/source-zh/rst_source/development/add_primitive.rst b/docs/source-zh/rst_source/development/add_primitive.rst index 909a7e82..6e81e6f9 100644 --- a/docs/source-zh/rst_source/development/add_primitive.rst +++ b/docs/source-zh/rst_source/development/add_primitive.rst @@ -120,10 +120,11 @@ primitives 方法，以及调用完成后的状态快照。区别仅在于方法 .. code-block:: python - def get_toolkit(*, primitives_kwargs, video_path=None): + def get_toolkit(*, primitives_kwargs, dashboard_events, video_path=None): from robots.myrobot.toolkit import MyRobotToolkit return MyRobotToolkit( primitives_kwargs=primitives_kwargs, + dashboard_events=dashboard_events, video_path=video_path, ) diff --git a/docs/source-zh/rst_source/development/add_robot.rst b/docs/source-zh/rst_source/development/add_robot.rst index 995a13e7..97e74413 100644 --- a/docs/source-zh/rst_source/development/add_robot.rst +++ b/docs/source-zh/rst_source/development/add_robot.rst @@ -17,8 +17,9 @@ RPent 的整体进程划分、服务职责和通信方式见 :doc:`系统设计 3. :ref:`定义 prompt `。 4. :ref:`实现 toolkit 和 primitives `。 5. :ref:`注册环境参数并生成 RunConfig `。 -6. 在 :ref:`_init_runtime ` 中启动或连接 ``env_server`` 与 - 所需的辅助服务。 +6. 实现 :ref:`runtime 钩子 `：普通 CLI 使用完整 runtime； + 如果环境支持 Dashboard 任务控制，再按 Session/TaskRun 生命周期实现仅供 + Dashboard 使用的拆分钩子。 .. _add-robot-entry: @@ -43,6 +44,7 @@ RPent 的整体进程划分、服务职责和通信方式见 :doc:`系统设计 .. code-block:: python # robots/myenv/__init__.py + from rpent.dashboard.events import DashboardEventSink from rpent.envs.env_spec import EnvSpec, RunConfig from rpent.envs.prompt_bundle import PromptBundle from robots.myenv.prompt_bundle import system_prompt, user_prompt @@ -53,12 +55,18 @@ RPent 的整体进程划分、服务职责和通信方式见 :doc:`系统设计 prompts=PromptBundle(system=system_prompt, user=user_prompt), add_cli_args=_add_cli_args, parse_config=_parse_config, + init_shared_runtime=_init_shared_runtime, + init_task_runtime=_init_task_runtime, init_runtime=_init_runtime, ) - def get_toolkit(*, primitives_kwargs, video_path=None): + def get_toolkit(*, primitives_kwargs, dashboard_events: DashboardEventSink, video_path=None): from robots.myenv.toolkit import MyEnvToolkit - return MyEnvToolkit(primitives_kwargs=primitives_kwargs, video_path=video_path) + return MyEnvToolkit( + primitives_kwargs=primitives_kwargs, + dashboard_events=dashboard_events, + video_path=video_path, + ) def _add_cli_args(parser, use_dashboard) -> None: """向共享 parser 注册环境参数。见第 4 节。""" @@ -68,18 +76,26 @@ RPent 的整体进程划分、服务职责和通信方式见 :doc:`系统设计 """校验最终的 args，返回 RunConfig。见第 4 节。""" ... - def _init_runtime(args, output_dir): - """启动 env_server、vla_server 及所需的辅助服务，构造 primitives_kwargs。 + def _init_runtime(args, output_dir, dashboard_events: DashboardEventSink): + """仅普通 CLI 使用：初始化完整 runtime。 返回 (daemons, primitives_kwargs)。见第 5 节。 """ ... + def _init_shared_runtime(args, output_dir, dashboard_events: DashboardEventSink): + """仅 Dashboard 使用：初始化由 Session 持有并复用的服务。""" + ... + + def _init_task_runtime(args, output_dir, dashboard_events: DashboardEventSink): + """仅 Dashboard 使用：为每个 TaskRun 初始化全新的服务。""" + ... + ``_resolve_env(name)`` 通过 ``importlib.import_module(f"robots.{name}")`` 动态加载环境包。因此，只需将环境包放在 ``robots/`` 下，无需维护中央注册列表。 下文依次说明这些模块需要实现的内容。``_add_cli_args`` 和 ``_parse_config`` -见第 4 节，``_init_runtime`` 见第 5 节。 +见第 4 节，三个 runtime 钩子见第 5 节。 .. _add-robot-env-rpc: @@ -253,13 +269,14 @@ primitives 的 ``__init__``。其中通常包含 **``_add_cli_args(parser, use_dashboard) -> None``。** 将环境参数注册到 main.py 已创建的共享 parser。``use_dashboard`` 决定原本必填的参数是否保持可选， -这些值随后由 Dashboard launcher 填入。main.py 会在 -``parser.parse_args()`` 之前调用该钩子，因此 argparse 的 usage 和错误信息也会 -包含环境参数。 +每个 Dashboard TaskRun 的 ``suite`` 与 ``task`` 会在 ``parse_config`` 调用前由 +``/rpent-task`` 命令提供。main.py 会在 ``parser.parse_args()`` 之前调用该钩子， +因此 argparse 的 usage 和错误信息也会包含环境参数。 -**``_parse_config(args) -> RunConfig``。** 在 ``parser.parse_args()`` 以及 -Dashboard launcher（如果启用）运行后调用。该钩子检查 Dashboard 模式下暂时设为 -可选的字段是否已经填入，并返回 :class:`~rpent.envs.RunConfig`： +**``_parse_config(args) -> RunConfig``。** 普通 CLI 模式下，该钩子在 +``parser.parse_args()`` 后调用；Dashboard 模式下，每个 TaskRun 会先把 +``/rpent-task`` 命令提供的 ``suite`` 与 ``task`` 写入任务参数，再调用该钩子。 +该钩子校验这些字段并返回 :class:`~rpent.envs.RunConfig`： - ``recipe_tag`` —— 单次运行的环境标签，用于 transcript 文件名和 recipe 路径 （LIBERO 使用 ``f"{suite.replace('libero_', '')}_t{task}_s{seed}"``）。 @@ -267,8 +284,6 @@ Dashboard launcher（如果启用）运行后调用。该钩子检查 Dashboard ``init_output_dir`` 创建目录并配置日志。 - ``prompt_vars`` —— 传给 ``PromptBundle.render`` 的字典，通常包含运行标识和 prompt 引用的其他变量。 -- ``dashboard_state`` —— ``args.dashboard`` 为真时是 - :class:`~rpent.dashboard.state.State`，否则为 ``None``。 - ``task_desc`` —— 环境特定的任务标识字典，会原样写入 transcript JSON 记录 （LIBERO 使用 ``{"suite": ..., "task": ..., "seed": ...}``）。 @@ -282,37 +297,42 @@ Dashboard launcher（如果启用）运行后调用。该钩子检查 Dashboard def _parse_config(args) -> RunConfig: if not args.suite: raise ValueError("--suite is required") - # ... 生成 recipe_tag、output_dir、prompt_vars、dashboard_state ... + # ... 生成 recipe_tag、output_dir 和 prompt_vars ... return RunConfig( recipe_tag=recipe_tag, output_dir=output_dir, prompt_vars=prompt_vars, - dashboard_state=dashboard_state, task_desc={"suite": args.suite, "task": args.task, "seed": args.seed}, ) .. _add-robot-runtime: -5. ``_init_runtime`` (runner 钩子) ----------------------------------- +5. Runtime 初始化钩子 +--------------------- + +三个 runtime 钩子都返回 ``(owned_daemons, primitives_kwargs)``： + +- ``owned_daemons: list[ProcessDaemon]`` 只包含当前进程实际启动的子进程， + 当前 runner 会在清理阶段停止它们。连接外部 endpoint 时，不能把外部服务加入 + 该列表。 +- ``primitives_kwargs: dict`` 会传给 toolkit 构造器，再由后者传入 primitives + 的 ``__init__``。完整参数通常包含 + ``{"env": MyEnvClient(...), "model": VLAClient(...)}``，以及其他辅助 client。 -``parse_config`` 返回后，main.py 调用 -``env_spec.init_runtime(args, output_dir)``，初始化环境与 VLA 服务，并构造 -toolkit 所需的参数。环境实现可以自行决定启动多少个子进程；当前 LIBERO 会启动 -``env_server``、``vla_server`` 和 ``sam3_server``。该钩子最终返回 -``(daemons, primitives_kwargs)``： +``init_runtime`` 是普通 CLI 钩子。``parse_config`` 返回后，``main.py`` 调用它 +一次，初始化完整 runtime。当前 LIBERO 实现会启动或连接 ``env_server``、 +``vla_server`` 和 ``sam3_server``，并一次性返回全部 primitive 参数。 -- ``daemons: list[ProcessDaemon]`` —— 本次运行拥有的子进程；main.py 在 - ``finally`` 里逐个 ``.stop()``。 -- ``primitives_kwargs: dict`` —— 原样传给 toolkit 构造器，再由后者传入 - primitives 的 ``__init__``。其中通常包含 - ``{"env": MyEnvClient(...), "model": VLAClient(...)}``；如果需要额外服务， - 也在这里加入相应的 client，例如 LIBERO 的 ``sam3_client``。 +``init_shared_runtime`` 和 ``init_task_runtime`` 是 **仅供 Dashboard 使用** 的 +钩子，普通 CLI 不会调用。Dashboard 会在每个 Session 中调用一次 +``init_shared_runtime``，初始化 Session 持有的复用服务；随后为每个全新的 +TaskRun 调用 ``init_task_runtime``，并合并两者返回的 ``primitives_kwargs``。 +具体如何拆分取决于环境自身的生命周期。LIBERO 将 VLA 和 SAM3 放在 Session +范围，将环境放在 TaskRun 范围；其他环境应采用适合自身服务的拆分，不必机械照搬。 endpoint（``--env-endpoint``、``--vla-endpoint``，以及 LIBERO 的 -``--sam3-endpoint``）解析和子进程启动（``--cuda-device`` 透传、 -``MUJOCO_GL``）也在这里完成，main.py 不处理这些细节。参考实现见 -``robots/libero/__init__.py``。 +``--sam3-endpoint``）解析、子进程启动和 runtime 状态事件，应放在拥有对应服务的 +钩子中，runner 不处理这些环境细节。参考实现见 ``robots/libero/__init__.py``。 冒烟测试 -------- diff --git a/docs/source-zh/rst_source/development/architecture.rst b/docs/source-zh/rst_source/development/architecture.rst index 54cfa2a6..da65fef2 100644 --- a/docs/source-zh/rst_source/development/architecture.rst +++ b/docs/source-zh/rst_source/development/architecture.rst @@ -91,28 +91,29 @@ Runner (``rpent/cli/main.py``) 2. 根据 ``args.env_name`` 调用 ``get_env_spec`` 加载环境定义，再通过 ``env_spec.add_cli_args(parser, use_dashboard=args.dashboard)`` 将该环境 的专用参数加入共享 parser。启用 Dashboard 时，原本必填的环境参数会暂时 - 设为可选，随后由配置页面填写。 + 设为可选，因为任务参数随后通过 Dashboard 命令提供。 3. 再调用 ``parser.parse_args()``，对完整参数集合执行 argparse 层的校验， 并生成最终的 ``args``；参数错误仍使用 argparse 的标准提示格式。 -4. 如果启用了 ``--dashboard``，启动配置页面，以当前参数作为默认值，并将 - 用户提交的配置写回 ``args``。 -5. 调用 ``env_spec.parse_config(args)`` 校验运行配置，并生成 +4. 如果启用了 ``--dashboard``，将控制权交给 ``rpent/cli/dashboard.py``，并在 + 长生命周期 Session 结束后返回。Dashboard 专用生命周期见下文；后续步骤属于 + 普通 CLI 路径。 +5. 调用 ``env_spec.parse_config(args)`` 校验普通 CLI 的运行配置，并生成 :class:`~rpent.envs.RunConfig`，其中包含 ``recipe_tag``、``output_dir``、 - ``prompt_vars``、``dashboard_state`` 和 ``task_desc``。启用 Dashboard - 时，此处还会确认配置页面已经补齐所需的环境参数。 + ``prompt_vars`` 和 ``task_desc``。 6. 调用 ``init_output_dir`` 创建本次运行的输出目录，并配置 ``run.log``。 7. 根据 ``--planner`` 调用 ``rpent.planner.base.build_planner`` 构造 **planner**，并使用环境提供的 prompt bundle 生成 system prompt 和 user prompt。 -8. 调用 ``env_spec.init_runtime(args, output_dir)``。环境实现会启动 - ``env_server`` 和 ``vla_server``；如果指定了 ``--env-endpoint`` 或 - ``--vla-endpoint``，则连接已有服务。该方法返回 +8. 调用 ``env_spec.init_runtime(args, output_dir, dashboard_events)``。环境实现会 + 启动 ``env_server``、``vla_server`` 和 ``sam3_server``；如果指定了对应 endpoint， + 则连接已有服务。该方法返回 ``(daemons, primitives_kwargs)``。 -9. 将 ``primitives_kwargs`` 传给环境的 ``get_toolkit`` 工厂，构造 - **toolkit**。 -10. 执行工具调用循环；启用 Dashboard 时，同时将运行事件发送到监控页面。 - 循环结束后保存 ``/transcript_*.json``，并在清理 toolkit - 时完成回合录像等收尾工作。 +9. 将 ``primitives_kwargs`` 和 ``dashboard_events`` 事件接收器传给环境的 + ``get_toolkit`` 工厂，构造 **toolkit**。一次性运行链路使用不执行任何 + 操作的事件接收器。 +10. 执行工具调用循环。循环结束后保存 + ``/transcript_*.json``，并在清理 toolkit 时完成回合录像等 + 收尾工作。 ``main.py`` 只负责连接上述步骤。环境相关实现集中在 ``robots//``， planner 后端集中在 ``rpent/planner/``， @@ -130,11 +131,12 @@ planner 后端集中在 ``rpent/planner/``， # robots/myenv/__init__.py def get_env_spec() -> EnvSpec: ... # 环境标识、提示词模板与 Runner 钩子 def get_toolkit( - *, primitives_kwargs, video_path=None, dashboard=None + *, primitives_kwargs, dashboard_events, video_path=None ): ... -``EnvSpec`` 汇集了环境的标识、prompt 模板与三个 Runner 钩子 -（``add_cli_args`` / ``parse_config`` / ``init_runtime``）；各字段要填什么见 +``EnvSpec`` 汇集了环境的标识、prompt 模板与五个 Runner 钩子： +``add_cli_args`` / ``parse_config`` / ``init_runtime``，以及仅供 Dashboard 使用的 +``init_shared_runtime`` / ``init_task_runtime``。各字段要填什么见 :doc:`interfaces`。 加载器本身不维护环境名称列表。当前 CLI 将 ``--env`` 限定为 ``libero`` @@ -157,19 +159,27 @@ Dashboard（可选） ----------------- ``rpent/dashboard/`` 由 FastAPI 应用和静态前端组成。启用 ``--dashboard`` 后， -``rpent/cli/main.py`` 会根据 ``--dashboard-host`` 和 ``--dashboard-port`` -启动 Dashboard；默认绑定 ``127.0.0.1``，并由操作系统分配可用端口。运行开始前， -用户可以先在配置页面确认或修改参数。 - -运行期间，Dashboard 页面提供： +``rpent/cli/main.py`` 会将控制权交给 ``rpent/cli/dashboard.py``，由后者根据 +``--dashboard-host`` 和 ``--dashboard-port`` 启动 Dashboard，并在启动共享服务前 +确认配置，然后调用一次仅供 Dashboard 使用的 +``env_spec.init_shared_runtime``。Session controller 随后等待 +``/rpent-task`` 命令；每次取得一个 TaskRun 后，Dashboard 会调用 +``parse_config`` 和仅供 Dashboard 使用的 ``env_spec.init_task_runtime``，合并 +共享与任务级 primitive 参数，并新建 toolkit 和 planner conversation。在 LIBERO +中，VLA 和 SAM3 会在 Dashboard 运行期间复用，每个 TaskRun 使用独立环境并按顺序 +执行。 + +TaskRun 运行期间，Dashboard 页面提供： - planner 输出以及工具调用事件； -- 实时相机画面和 Pi0.5 视图； +- 实时固定相机和腕部相机画面； - 动作时间线和单步动作片段； - 运行结束后的完整回合录像（如果已生成）。 +页面可以提交普通 planner 消息、新任务命令和中断请求，但不会直接发出机器人 +动作。planner、toolkit 和环境运行时通过 ``dashboard_events`` 事件接收器发布 +展示更新。 服务端通过 SSE 推送运行状态摘要，前端再按需读取详细事件、时间线和图像。 -Dashboard 使用 planner 与 toolkit 产生的状态进行展示，不直接发出机器人动作。 下一步 ------ diff --git a/docs/source-zh/rst_source/development/interfaces.rst b/docs/source-zh/rst_source/development/interfaces.rst index d28c3d34..642785c3 100644 --- a/docs/source-zh/rst_source/development/interfaces.rst +++ b/docs/source-zh/rst_source/development/interfaces.rst @@ -12,7 +12,7 @@ .. code-block:: python def get_env_spec() -> EnvSpec: ... - def get_toolkit(*, primitives_kwargs, video_path=None, dashboard=None): ... + def get_toolkit(*, primitives_kwargs, dashboard_events: DashboardEventSink, video_path=None): ... ``get_env_spec`` 返回 ``EnvSpec``，其中你需要提供： @@ -33,11 +33,18 @@ - 校验参数并返回 ``RunConfig``；``recipe_tag``、``output_dir``、``prompt_vars`` 三项需由你正确填写（供 prompt 模板插值）。 * - ``init_runtime`` - - 启动或连接 env 与 VLA 等子进程，构造 ``primitives_kwargs`` 字典 - （env 客户端、模型客户端等），供 toolkit 组装 primitives。 - -``get_toolkit`` 一般只需把 ``primitives_kwargs`` 传给环境子类；``video_path``、 - ``dashboard`` 由 ``main.py`` 传入，通常不用改。 + - 仅普通 CLI 使用：启动或连接完整 runtime，构造 ``primitives_kwargs`` + 字典（env 客户端、模型客户端等），供 toolkit 组装 primitives； + ``DashboardEventSink`` 用于上报运行时状态。 + * - ``init_shared_runtime`` + - 仅 Dashboard 使用：初始化可供多个 TaskRun 复用、由 Session 持有的服务， + 并返回其本地 daemon 与 primitive 参数。 + * - ``init_task_runtime`` + - 仅 Dashboard 使用：为每个 TaskRun 初始化全新的任务级服务，并返回其本地 + daemon 与 primitive 参数。 + +``get_toolkit`` 一般只需把 ``primitives_kwargs`` 传给环境子类； +``dashboard_events``、``video_path`` 由当前 runner 传入，通常不用改。 参考实现：``robots/libero/__init__.py``。 @@ -58,6 +65,7 @@ Planner toolkit: Toolkit, max_turns: int, input_queue=None, + dashboard_interaction=None, ) -> PlannerResult: ... 约定：用 ``toolkit.get_tools_spec()`` 把工具交给模型；每次调用 ``toolkit.execute_tool(name, input_dict)``； @@ -94,7 +102,8 @@ Planner 接已有server或写 ``env_server`` / ``vla_server`` 时关注下面两点。 -客户端端点（在 ``add_cli_args`` 里暴露，或在 ``init_runtime`` 里解析）： +客户端端点（在 ``add_cli_args`` 中暴露，并在适用的普通 CLI 或 Dashboard +runtime 钩子中解析）： .. code-block:: text diff --git a/docs/source-zh/rst_source/quickstart.rst b/docs/source-zh/rst_source/quickstart.rst index b9cf71c7..faf6ee0f 100644 --- a/docs/source-zh/rst_source/quickstart.rst +++ b/docs/source-zh/rst_source/quickstart.rst @@ -42,14 +42,18 @@ LIBERO-PRO 仿真资源。下面以 LIBERO-PRO 和 ``claude_code`` planner 3. 通过 Dashboard 查看运行过程 ------------------------------ -添加 ``--dashboard`` 后，RPent 会启动本地 Dashboard 服务，并在终端输出访问地址。打开该地址后，可以先在启动页面确认配置。运行开始后，Dashboard 会实时显示智能体的推理过程、相机与 Pi0 视图、动作时间线和片段回放。使用 ``--dashboard-language zh-cn`` 可切换到中文界面。 +添加 ``--dashboard`` 后，RPent 会启动本地 Dashboard，并在终端输出访问地址： .. code-block:: bash rpent --env libero --dashboard --dashboard-language zh-cn \ - --suite libero_object_swap --task 2 --seed 0 \ --planner claude_code --model claude-opus-4-8 +打开该地址并确认配置；服务就绪后，在页面输入 +``/rpent-task libero_object_swap 2 0`` 启动任务。Dashboard 会实时显示智能体的 +推理过程、相机画面和动作时间线；任务结束后可以继续提交下一任务。使用 +``--dashboard-language zh-cn`` 可切换到中文界面。 + 关键 CLI 选项 ------------- diff --git a/docs/source-zh/rst_source/usage/advanced_deployment.rst b/docs/source-zh/rst_source/usage/advanced_deployment.rst index 63e7fab4..8060fbd7 100644 --- a/docs/source-zh/rst_source/usage/advanced_deployment.rst +++ b/docs/source-zh/rst_source/usage/advanced_deployment.rst @@ -10,6 +10,9 @@ Pi0.5 VLA 用 ``--vla-endpoint``，SAM3 用 ``--sam3-endpoint``。每个都取 ``[protocol://]HOST:PORT``，省略 protocol 时默认 HTTP，也可用 ``socket://`` 改走 socket RPC。 +Dashboard Session 不支持 ``--env-endpoint``，因为每个 TaskRun 都需要使用新的 +环境服务；Dashboard 模式仍可使用 ``--vla-endpoint`` 和 ``--sam3-endpoint``。 + LIBERO 环境服务 --------------- diff --git a/docs/source-zh/rst_source/usage/libero.rst b/docs/source-zh/rst_source/usage/libero.rst index 2675458b..28cb9fa5 100644 --- a/docs/source-zh/rst_source/usage/libero.rst +++ b/docs/source-zh/rst_source/usage/libero.rst @@ -150,17 +150,30 @@ LIBERO 工具分为物理动作工具和只读工具。 Dashboard --------- -加上 ``--dashboard`` 可启动本地监控服务。系统会自动选择一个空闲端口， -并在终端输出访问 URL： +加上 ``--dashboard`` 可启动长生命周期的本地 Dashboard Session。系统会自动 +选择一个空闲端口，并在终端输出访问 URL： .. code-block:: bash rpent --env libero --dashboard \ - --suite libero_object_swap --task 2 --seed 0 \ --planner claude_code --model claude-opus-4-8 -Dashboard 会实时展示推理过程、agentview 视图、腕部相机视图、Pi0.5 -叠加信息和动作时间线。使用 ``--dashboard-language zh-cn`` 切换中文 UI。 +打开该地址，确认 Session 配置并点击 **Start Session**。共享服务就绪后，在页面 +输入以下命令启动 TaskRun： + +.. code-block:: text + + /rpent-task libero_object_swap 2 0 + +每个 TaskRun 都使用新的环境，VLA 和 SAM3 服务则由 Session 复用。任务完成后可以 +继续提交另一条 ``/rpent-task``；任务运行期间提交新命令则会切换任务。Dashboard +会实时展示智能体的推理和工具调用、固定相机与腕部相机视图、动作时间线，以及动作 +片段和完整 episode 回放。TaskRun 运行时，输入普通消息可以引导智能体，按 Esc +可以请求中断。在终端按 Ctrl+C 可结束整个 Session。 + +``--dashboard`` 不能与 ``--interactive`` 或 ``--env-endpoint`` 同时使用；外部 +``--vla-endpoint`` 和 ``--sam3-endpoint`` 服务仍然可用。使用 +``--dashboard-language zh-cn`` 可切换中文 UI。 接入自定义 VLA ----------------