Skip to content

Support Multi-Backend for Demo Scripts #5938

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ooctipus merged 12 commits into from
Jun 7, 2026
Merged

Support Multi-Backend for Demo Scripts #5938

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
367804d
Support Multi-backend for Demo Scripts
Jun 2, 2026
20100eb
Unify Code Format
Jun 3, 2026
49c5927
Improve Viz
Jun 3, 2026
0679143
Lint
Jun 3, 2026
0b7e5eb
Reorder Imports
Jun 3, 2026
cc2b00b
Update Newton Config for hands.py
Jun 4, 2026
02f996c
Fix Shadow Hand Tendons
Jun 4, 2026
28fb841
Use New API for quadcopter.py
Jun 4, 2026
e822c26
Remove Config Override For Shadow Hand
Jun 5, 2026
a93be33
Simplify quadrupeds.py
Jun 5, 2026
13ec175
Remove string_to_callable
Jun 6, 2026
b0847bc
Add Change Logs
Jun 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
116 changes: 64 additions & 52 deletions scripts/demos/arms.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,53 +8,55 @@

.. code-block:: bash

# Usage
# Usage with default PhysX physics and default kit visualizer.
./isaaclab.sh -p scripts/demos/arms.py

# Usage with Newton visualizer and default PhysX physics.
./isaaclab.sh -p scripts/demos/arms.py --visualizer newton

# Usage with Newton (MJWarp) physics and default kit visualizer.
./isaaclab.sh -p scripts/demos/arms.py --physics newton_mjwarp

# Usage with Newton visualizer and Newton (MJWarp) physics.
./isaaclab.sh -p scripts/demos/arms.py --visualizer newton --physics newton_mjwarp

"""

"""Launch Isaac Sim Simulator first."""
"""Parse CLI first so we can decide whether to launch Isaac Sim Kit."""

import argparse
from typing import TYPE_CHECKING

from isaaclab.app import AppLauncher
from isaaclab.app import add_launcher_args, launch_simulation

# add argparse arguments
parser = argparse.ArgumentParser(description="This script demonstrates different single-arm manipulators.")
# append AppLauncher cli args
AppLauncher.add_app_launcher_args(parser)
# demos should open Kit visualizer by default
parser = argparse.ArgumentParser(
description="This script demonstrates different single-arm manipulators.",
conflict_handler="resolve",
)
parser.add_argument("--physics", default="physx", choices=["physx", "newton_mjwarp"], help="Physics backend.")
add_launcher_args(parser)
parser.set_defaults(visualizer=["kit"])
# parse the arguments
args_cli = parser.parse_args()

# launch omniverse app
app_launcher = AppLauncher(args_cli)
simulation_app = app_launcher.app

"""Rest everything follows."""

import numpy as np
import torch

import isaaclab.sim as sim_utils
from isaaclab.assets import Articulation
from isaaclab.utils.assets import ISAAC_NUCLEUS_DIR

##
# Pre-defined configs
##
# isort: off
from isaaclab_assets import (
FRANKA_PANDA_CFG,
UR10_CFG,
KINOVA_JACO2_N7S300_CFG,
KINOVA_JACO2_N6S300_CFG,
KINOVA_GEN3_N7_CFG,
SAWYER_CFG,
)
from isaaclab.physics import PhysicsCfg
from isaaclab.utils.assets import ISAAC_NUCLEUS_DIR

from isaaclab_newton.physics import MJWarpSolverCfg, NewtonCfg # isort:skip
from isaaclab_assets.robots.franka import FRANKA_PANDA_CFG # isort:skip
from isaaclab_assets.robots.kinova import KINOVA_GEN3_N7_CFG, KINOVA_JACO2_N6S300_CFG, KINOVA_JACO2_N7S300_CFG # isort:skip
from isaaclab_assets.robots.sawyer import SAWYER_CFG # isort:skip
from isaaclab_assets.robots.universal_robots import UR10_CFG # isort:skip

# isort: on
if TYPE_CHECKING:
from isaaclab.assets import Articulation


def define_origins(num_origins: int, spacing: float) -> list[list[float]]:
Expand Down Expand Up @@ -93,7 +95,7 @@ def design_scene() -> tuple[dict, list[list[float]]]:
# -- Robot
franka_arm_cfg = FRANKA_PANDA_CFG.replace(prim_path="/World/Origin1/Robot")
franka_arm_cfg.init_state.pos = (0.0, 0.0, 1.05)
franka_panda = Articulation(cfg=franka_arm_cfg)
franka_panda = franka_arm_cfg.class_type(franka_arm_cfg)

# Origin 2 with UR10
sim_utils.create_prim("/World/Origin2", "Xform", translation=origins[1])
Expand All @@ -105,7 +107,7 @@ def design_scene() -> tuple[dict, list[list[float]]]:
# -- Robot
ur10_cfg = UR10_CFG.replace(prim_path="/World/Origin2/Robot")
ur10_cfg.init_state.pos = (0.0, 0.0, 1.03)
ur10 = Articulation(cfg=ur10_cfg)
ur10 = ur10_cfg.class_type(ur10_cfg)

# Origin 3 with Kinova JACO2 (7-Dof) arm
sim_utils.create_prim("/World/Origin3", "Xform", translation=origins[2])
Expand All @@ -115,7 +117,7 @@ def design_scene() -> tuple[dict, list[list[float]]]:
# -- Robot
kinova_arm_cfg = KINOVA_JACO2_N7S300_CFG.replace(prim_path="/World/Origin3/Robot")
kinova_arm_cfg.init_state.pos = (0.0, 0.0, 0.8)
kinova_j2n7s300 = Articulation(cfg=kinova_arm_cfg)
kinova_j2n7s300 = kinova_arm_cfg.class_type(kinova_arm_cfg)

# Origin 4 with Kinova JACO2 (6-Dof) arm
sim_utils.create_prim("/World/Origin4", "Xform", translation=origins[3])
Expand All @@ -125,7 +127,7 @@ def design_scene() -> tuple[dict, list[list[float]]]:
# -- Robot
kinova_arm_cfg = KINOVA_JACO2_N6S300_CFG.replace(prim_path="/World/Origin4/Robot")
kinova_arm_cfg.init_state.pos = (0.0, 0.0, 0.8)
kinova_j2n6s300 = Articulation(cfg=kinova_arm_cfg)
kinova_j2n6s300 = kinova_arm_cfg.class_type(kinova_arm_cfg)

# Origin 5 with Sawyer
sim_utils.create_prim("/World/Origin5", "Xform", translation=origins[4])
Expand All @@ -135,7 +137,7 @@ def design_scene() -> tuple[dict, list[list[float]]]:
# -- Robot
kinova_arm_cfg = KINOVA_GEN3_N7_CFG.replace(prim_path="/World/Origin5/Robot")
kinova_arm_cfg.init_state.pos = (0.0, 0.0, 1.05)
kinova_gen3n7 = Articulation(cfg=kinova_arm_cfg)
kinova_gen3n7 = kinova_arm_cfg.class_type(kinova_arm_cfg)

# Origin 6 with Kinova Gen3 (7-Dof) arm
sim_utils.create_prim("/World/Origin6", "Xform", translation=origins[5])
Expand All @@ -147,7 +149,7 @@ def design_scene() -> tuple[dict, list[list[float]]]:
# -- Robot
sawyer_arm_cfg = SAWYER_CFG.replace(prim_path="/World/Origin6/Robot")
sawyer_arm_cfg.init_state.pos = (0.0, 0.0, 1.03)
sawyer = Articulation(cfg=sawyer_arm_cfg)
sawyer = sawyer_arm_cfg.class_type(sawyer_arm_cfg)

# return the scene information
scene_entities = {
Expand All @@ -161,14 +163,14 @@ def design_scene() -> tuple[dict, list[list[float]]]:
return scene_entities, origins


def run_simulator(sim: sim_utils.SimulationContext, entities: dict[str, Articulation], origins: torch.Tensor):
def run_simulator(sim: "sim_utils.SimulationContext", entities: dict[str, "Articulation"], origins: torch.Tensor):
"""Runs the simulation loop."""
# Define simulation stepping
sim_dt = sim.get_physics_dt()
sim_time = 0.0
count = 0
# Simulate physics
while simulation_app.is_running():
# Step while a visualizer window is still open (or none exist, e.g. headless); works for kit and newton.
while sim.is_headless_or_exist_active_visualizer():
# reset
if count % 200 == 0:
# reset counters
Expand Down Expand Up @@ -214,24 +216,34 @@ def run_simulator(sim: sim_utils.SimulationContext, entities: dict[str, Articula

def main():
"""Main function."""
# Initialize the simulation context
sim_cfg = sim_utils.SimulationCfg(device=args_cli.device)
sim = sim_utils.SimulationContext(sim_cfg)
# Set main camera
sim.set_camera_view([3.5, 0.0, 3.2], [0.0, 0.0, 0.5])
# design scene
scene_entities, scene_origins = design_scene()
scene_origins = torch.tensor(scene_origins, device=sim.device)
# Play the simulator
sim.reset()
# Now we are ready!
print("[INFO]: Setup complete...")
# Run the simulator
run_simulator(sim, scene_entities, scene_origins)
with launch_simulation(cfg=PhysicsCfg(), launcher_args=args_cli) as physics_cfg:
# The default newton mjwarp solver configuration needs to be tuned for these arms.
if isinstance(physics_cfg, NewtonCfg) and isinstance(physics_cfg.solver_cfg, MJWarpSolverCfg):
physics_cfg.solver_cfg.njmax = 70
physics_cfg.solver_cfg.nconmax = 70
physics_cfg.solver_cfg.ls_iterations = 40
physics_cfg.solver_cfg.cone = "elliptic"
physics_cfg.solver_cfg.impratio = 100

@isaaclab-review-bot isaaclab-review-bot Bot Jun 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 Low: Duplicated solver tuning configuration

This identical MJWarp solver configuration block (njmax, nconmax, ls_iterations, cone, impratio, ls_parallel, integrator, num_substeps) is repeated verbatim across arms.py, bipeds.py, and hands.py.

Consider extracting a shared helper or preset:

# In a shared module, e.g. isaaclab_assets.newton_presets
ARTICULATION_MJWARP_PRESET = dict(
    njmax=70, nconmax=70, ls_iterations=40,
    cone="elliptic", impratio=100,
    ls_parallel=False, integrator="implicitfast",
)

This reduces the maintenance burden when solver defaults change and makes it obvious which demos share physics characteristics.

@YizeWang YizeWang Jun 3, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, we want to expose the way to tune parameters.

@isaaclab-review-bot isaaclab-review-bot Bot Jun 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That makes sense — for demo scripts, having the tuning parameters explicit and self-contained is more instructive for users learning how to configure the solver. Keeping each demo standalone avoids an extra indirection that could obscure the workflow.

No objection from me. Thanks for clarifying!

physics_cfg.solver_cfg.ls_parallel = False
physics_cfg.solver_cfg.integrator = "implicitfast"
physics_cfg.num_substeps = 2

# Initialize the simulation context
sim_cfg = sim_utils.SimulationCfg(device=args_cli.device, physics=physics_cfg)
sim = sim_utils.SimulationContext(sim_cfg)
# Set main camera
sim.set_camera_view([3.5, 0.0, 3.2], [0.0, 0.0, 0.5])
# design scene
scene_entities, scene_origins = design_scene()
scene_origins = torch.tensor(scene_origins, device=sim.device)
# Play the simulator
sim.reset()
# Now we are ready!
print("[INFO]: Setup complete...")
# Run the simulator
run_simulator(sim, scene_entities, scene_origins)


if __name__ == "__main__":
# run the main function
main()
# close sim app
simulation_app.close()
100 changes: 58 additions & 42 deletions scripts/demos/bin_packing.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,49 +13,60 @@

.. code-block:: bash

# Usage
./isaaclab.sh -p scripts/demos/bin_packing.py --num_envs 32
# Usage with default PhysX physics and default kit visualizer.
./isaaclab.sh -p scripts/demos/bin_packing.py

# Usage with Newton visualizer and default PhysX physics.
./isaaclab.sh -p scripts/demos/bin_packing.py --visualizer newton

# Usage with Newton (MJWarp) physics and default kit visualizer.
./isaaclab.sh -p scripts/demos/bin_packing.py --physics newton_mjwarp

# Usage with Newton visualizer and Newton (MJWarp) physics.
./isaaclab.sh -p scripts/demos/bin_packing.py --visualizer newton --physics newton_mjwarp

"""

from __future__ import annotations

"""Launch Isaac Sim Simulator first."""

"""Parse CLI first so we can decide whether to launch Isaac Sim Kit."""

import argparse
from typing import TYPE_CHECKING

from isaaclab.app import AppLauncher
from isaaclab.app import add_launcher_args, launch_simulation

# add argparse arguments
parser = argparse.ArgumentParser(description="Demo usage of RigidObjectCollection through bin packing example")
parser = argparse.ArgumentParser(
description="Demo usage of RigidObjectCollection through bin packing example",
conflict_handler="resolve",
)
parser.add_argument("--num_envs", type=int, default=16, help="Number of environments to spawn.")
# append AppLauncher cli args
AppLauncher.add_app_launcher_args(parser)
# demos should open Kit visualizer by default
parser.add_argument("--physics", default="physx", choices=["physx", "newton_mjwarp"], help="Physics backend.")
add_launcher_args(parser)
parser.set_defaults(visualizer=["kit"])
# parse the arguments
args_cli = parser.parse_args()

# launch omniverse app
app_launcher = AppLauncher(args_cli)
simulation_app = app_launcher.app

"""Rest everything follows."""

import math

import torch

##
# Pre-defined configs
##
from isaaclab_newton.physics import MJWarpSolverCfg, NewtonCfg

import isaaclab.sim as sim_utils
import isaaclab.utils.math as math_utils
from isaaclab.assets import AssetBaseCfg, RigidObjectCfg, RigidObjectCollection, RigidObjectCollectionCfg
from isaaclab.assets import AssetBaseCfg, RigidObjectCfg, RigidObjectCollectionCfg
Comment thread
YizeWang marked this conversation as resolved.
from isaaclab.physics import PhysicsCfg
from isaaclab.scene import InteractiveScene, InteractiveSceneCfg
from isaaclab.sim import SimulationContext
from isaaclab.utils import Timer
from isaaclab.utils.assets import ISAAC_NUCLEUS_DIR
from isaaclab.utils.configclass import configclass

if TYPE_CHECKING:
from isaaclab.assets import RigidObjectCollection

##
# Scene Configuration
##
Expand Down Expand Up @@ -131,7 +142,7 @@ class MultiObjectSceneCfg(InteractiveSceneCfg):
# Instantiate four grocery variants per layer and replicate across all layers in each environment.
rigid_objects={
f"Object_{label}_Layer{layer}": RigidObjectCfg(
prim_path=f"/World/envs/env_.*/Object_{label}_Layer{layer}",
prim_path=f"/World/envs/env_.*/Groceries/Object_{label}_Layer{layer}",

@isaaclab-review-bot isaaclab-review-bot Bot Jun 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟠 Low: Prim path hierarchy change

The prim path changed from /World/envs/env_.*/Object_{label}_Layer{layer} to /World/envs/env_.*/Groceries/Object_{label}_Layer{layer} (added Groceries/ intermediate Xform).

This is fine for a fresh run, but could break:

  • Saved USD stages that reference the old paths
  • Any downstream tooling or tests that pattern-match on prim paths

Is there a corresponding change that creates the /Groceries Xform prim? I don't see one in this diff — the InteractiveScene may handle it automatically, but it's worth confirming.

@YizeWang YizeWang Jun 3, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Newton visualizer needs this change. Otherwise, objects cannot be properly visualized.

@isaaclab-review-bot isaaclab-review-bot Bot Jun 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clarification! That makes sense — if Newton visualizer requires this hierarchy for proper object rendering, then this is the right approach. The InteractiveScene should handle creating the intermediate Xform automatically based on the prim path.

Good to go 👍

@isaaclab-review-bot isaaclab-review-bot Bot Jun 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clarification! That makes sense — the Newton visualizer likely needs the intermediate /Groceries Xform to correctly traverse and render the child prims in its scene graph.

Since this is a new demo script (not modifying an existing one), there are no pre-existing saved stages or path references to worry about. My concern about backward compatibility doesn't apply here. 👍

No further concerns on this change.

init_state=RigidObjectCfg.InitialStateCfg(pos=(x, y, 0.2 + (layer) * 0.2)),
spawn=GROCERIES.get(f"OBJECT_{label}"),
)
Expand Down Expand Up @@ -263,7 +274,7 @@ def build_grocery_defaults(
##


def run_simulator(sim: SimulationContext, scene: InteractiveScene) -> None:
def run_simulator(sim: sim_utils.SimulationContext, scene: InteractiveScene) -> None:
"""Runs the simulation loop that coordinates spawn randomization and stepping.

Returns:
Expand Down Expand Up @@ -295,8 +306,8 @@ def run_simulator(sim: SimulationContext, scene: InteractiveScene) -> None:
# Precompute a helper mask to toggle objects between active and cached sets.
# Precompute XY bounds [[x_min,y_min],[x_max,y_max]]
bounds_xy = torch.as_tensor(BIN_XY_BOUND, device=device, dtype=spawn_poses_w.dtype)
# Simulation loop
while simulation_app.is_running():
# Step while a visualizer window is still open (or none exist, e.g. headless); works for kit and newton.
while sim.is_headless_or_exist_active_visualizer():
# Reset
if count % 250 == 0:
# reset counter
Expand Down Expand Up @@ -343,27 +354,32 @@ def main() -> None:
Returns:
None: The function drives the simulation for its side-effects.
"""
# Load kit helper
sim_cfg = sim_utils.SimulationCfg(dt=0.005, device=args_cli.device)
sim = SimulationContext(sim_cfg)
# Set main camera
sim.set_camera_view((2.5, 0.0, 4.0), (0.0, 0.0, 2.0))

# Design scene
scene_cfg = MultiObjectSceneCfg(num_envs=args_cli.num_envs, env_spacing=1.0, replicate_physics=True)
with Timer("[INFO] Time to create scene: "):
scene = InteractiveScene(scene_cfg)

# Play the simulator
sim.reset()
# Now we are ready!
print("[INFO]: Setup complete...")
# Run the simulator
run_simulator(sim, scene)
with launch_simulation(cfg=PhysicsCfg(), launcher_args=args_cli) as physics_cfg:
# The default newton mjwarp solver configuration needs to be tuned for this demo.
if isinstance(physics_cfg, NewtonCfg) and isinstance(physics_cfg.solver_cfg, MJWarpSolverCfg):
physics_cfg.solver_cfg.nconmax = 128
physics_cfg.solver_cfg.naconmax = 2048
physics_cfg.solver_cfg.njmax = 512

# Load kit helper
sim_cfg = sim_utils.SimulationCfg(dt=0.005, device=args_cli.device, physics=physics_cfg)
sim = sim_utils.SimulationContext(sim_cfg)
# Set main camera
sim.set_camera_view((2.5, 0.0, 4.0), (0.0, 0.0, 2.0))

# Design scene
scene_cfg = MultiObjectSceneCfg(num_envs=args_cli.num_envs, env_spacing=1.0, replicate_physics=True)
with Timer("[INFO] Time to create scene: "):
scene = scene_cfg.class_type(scene_cfg)

# Play the simulator
sim.reset()
# Now we are ready!
print("[INFO]: Setup complete...")
# Run the simulator

@isaaclab-review-bot isaaclab-review-bot Bot Jun 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium: Indirect construction via string_to_callable

Using string_to_callable("isaaclab.scene:InteractiveScene")(scene_cfg) here is unusual compared to the other demos which import and call classes directly. This pattern:

  1. Obscures the dependency — readers won't immediately see what's being constructed.
  2. Bypasses static analysis / IDE navigation.
  3. Differs from the approach in every other demo file in this PR.

Is there a specific reason InteractiveScene cannot be imported directly here (e.g., circular import when Newton backend is active)? If so, a brief comment explaining why would help future maintainers. If not, consider using the direct import for consistency:

from isaaclab.scene import InteractiveScene
scene = InteractiveScene(scene_cfg)

@YizeWang YizeWang Jun 3, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ooctipus Could you please take a look here? I tried to avoid local imports. Not sure if there exists a better way to achieve.

@isaaclab-review-bot isaaclab-review-bot Bot Jun 3, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for raising this @YizeWang — happy to share my thoughts while you wait for @ooctipus.

Looking at the file, InteractiveScene is actually already directly imported at the top:

from isaaclab.scene import InteractiveScene, InteractiveSceneCfg

So unless there's a runtime reason the import needs to be deferred (e.g., the class identity changes depending on the active physics backend), you could simply use:

scene = InteractiveScene(scene_cfg)

…which would be consistent with the existing import and easier to follow.

If there is a backend-aware dispatch happening (where "isaaclab.scene:InteractiveScene" might resolve to a different class at runtime depending on the physics context), then string_to_callable makes sense — but a short inline comment explaining why would really help future readers. Something like:

# Use string_to_callable to allow backend-specific scene class resolution at runtime.
scene = string_to_callable("isaaclab.scene:InteractiveScene")(scene_cfg)

Deferring to @ooctipus on whether the multi-backend machinery requires this indirection here.

@YizeWang YizeWang Jun 6, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Improved in the latest commit "Remove string_to_callable"

@isaaclab-review-bot isaaclab-review-bot Bot Jun 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great, thanks for removing it! 👍

@isaaclab-review-bot isaaclab-review-bot Bot Jun 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, using scene_cfg.class_type(scene_cfg) is even cleaner than my suggestion — it keeps the construction tied to the config definition. Thanks for addressing this! 👍

run_simulator(sim, scene)


if __name__ == "__main__":
# run the main execution
main()
# close sim app
simulation_app.close()
Loading
Loading