[DRAFT][Newton][Factory] NutThread support#5836
Draft
mzamoramora-nvidia wants to merge 37 commits into
Draft
Conversation
Add NewtonCollisionPipelineCfg and HydroelasticSDFCfg to expose Newton's collision pipeline parameters (broad phase, contact limits, hydroelastic mode) through NewtonCfg.collision_cfg. Add MJWarpSolverCfg.tolerance for solver convergence control. Fix validation order so collision_cfg is stored before the use_mujoco_contacts consistency check runs. Reset _collision_cfg in clear() to avoid stale state across reset cycles. Fall back to a default CollisionPipeline when collision_cfg is None.
Move config resolution out of NewtonManager into NewtonCollisionPipelineCfg.to_pipeline_args(), following the Kamino to_solver_config() pattern. Fix truthiness check on hydroelastic config dict to use explicit `is not None`. Add missing changelog entry for MJWarpSolverCfg.tolerance. Use specific return type hint dict[str, Any].
Add SDFCfg @configclass for SDF mesh collision configuration with fields for resolution, narrow band, margin, body/shape pattern matching, per-pattern resolution overrides, visual mesh fallback, and hydroelastic stiffness assignment. Add five missing fields to HydroelasticSDFCfg: moment_matching, buffer_mult_broad, buffer_mult_iso, buffer_mult_contact, and grid_size.
Add sdf_cfg field (SDFCfg | None) to NewtonCfg and re-export SDFCfg from the physics package __init__.pyi so callers can configure SDF mesh collision at the top-level manager config.
Add _build_sdf_on_mesh, _create_sdf_collision_from_visual, and _apply_sdf_config classmethods to NewtonManager. These methods apply SDFCfg settings to matching mesh shapes in the model builder, including per-pattern resolution overrides and optional hydroelastic flags.
When SDF patterns are configured, exclude matching mesh shapes from convex-hull approximation so their triangle geometry is preserved for mesh.build_sdf(). Also call NewtonManager._apply_sdf_config() on each prototype before add_builder copies it N times, so SDF is built once and all environments inherit it.
Unit tests for NewtonManager._build_sdf_on_mesh and _apply_sdf_config using unittest.mock to avoid needing a running Newton simulation. Covers resolution overrides, hydroelastic flags, body/shape pattern matching, and non-mesh shape exclusion.
Fix max_resolution/target_voxel_size docstrings to accurately describe that both are forwarded to Newton. Remove unreachable collision_cfg force-override block in initialize_solver.
- Pass hydro_patterns to _create_sdf_collision_from_visual so it respects hydroelastic_shape_patterns filter (was unconditionally setting HYDROELASTIC) - Replace getattr(cfg, "sdf_cfg", None) with direct attribute access in all 3 locations - Add regex validation with actionable ValueError on invalid patterns - Log warning and return False from _build_sdf_on_mesh when mesh is None; only count successful SDF builds in num_patched - Fix :attr: -> :class: for SDFCfg reference in docstring
## Summary - Adds ``NewtonCfg.collision_decimation`` (default ``0`` = legacy one-collide-per-tick behaviour). - When positive, ``_run_solver_substeps`` re-invokes the collision pipeline every ``collision_decimation`` substeps so contact normals reflect the bodies' just-integrated poses instead of the poses from the top of the tick. - The last substep is intentionally skipped — its contact set would only affect the next tick, which the top-of-tick ``collide()`` in ``_simulate_physics_only`` / ``_simulate_full`` owns. - ``NewtonCfg.__post_init__`` warns when ``collision_decimation >= num_substeps`` (silently equivalent to ``0``). ## Test plan - [x] ``./isaaclab.sh -p -m pytest source/isaaclab_newton/test/physics/test_newton_manager_abstraction.py`` — 39/39 pass - [x] ``./isaaclab.sh -f`` — pre-commit clean
## Summary - Adds ``test_newton_cfg_collision_decimation_warning`` (parametrized over ``(num_substeps, collision_decimation)``) for the ``__post_init__`` warning gate. - Adds ``test_collision_decimation_invokes_mid_loop_collide`` that wraps ``NewtonManager._collision_pipeline.collide`` with a counter, runs one physics tick with a sphere falling onto a ground plane, and asserts the call count equals ``1 + floor((num_substeps - 1) / collision_decimation)``. - Adds ``source/isaaclab_newton/changelog.d/mzamoramora-collision-decimation.minor.rst``. ## Test plan - [x] ``./isaaclab.sh -p -m pytest source/isaaclab_newton/test/physics/test_newton_manager_abstraction.py`` — 39/39 pass (28 existing + 11 new) - [x] ``./isaaclab.sh -f`` — pre-commit clean
…nfiguration Merged Antoine's antoiner/newton-sdf-config branch with conflict resolution: - extension.toml / CHANGELOG.rst: kept HEAD (develop has advanced past PR's manual version bumps; changelog.d conventions supersede manual edits). - __init__.pyi: union of exports — kept develop's manager classes + added SDFCfg. - newton_collision_cfg.py: union — kept develop's import/doc updates, added back PR's 5 hydroelastic fields and the SDFCfg class. - newton_replicate.py: union — kept develop's deformable-prim handling hooks, layered PR's SDF-pattern skip logic and _apply_sdf_config call. - newton_manager.py: kept develop's structure (modernized API), surgically grafted PR's three SDF helper methods (_build_sdf_on_mesh, _create_sdf_collision_from_visual, _apply_sdf_config) and 're' import. - newton_manager_cfg.py: kept develop's structure, added sdf_cfg field on NewtonCfg and SDFCfg import. Not carried over: PR's initialize_solver SDF-detection branch and _initialize_contacts hydroelastic warning (deferred — develop's initialize_solver has been refactored to delegate to _build_solver). This vendoring is for local use on the bundle branch. When PR isaac-sim#5228 lands upstream, this merge commit is reverted and develop is pulled in.
…tick re-collide
``RigidObjectData._create_simulation_bindings`` crashed with ``IndexError: tuple index out of range`` whenever the rigid object was fixed-base AND its root view only contained a single body — e.g. a kinematic-enabled rigid object built from a USD that authors a ``PhysicsFixedJoint``. Newton's ``get_root_transforms`` returns a 2D ``(count, links)`` array in that scenario, but the legacy ``is_fixed_base`` branch unconditionally indexed ``[:, 0, 0]`` assuming a 3D ``(count, links, 1)`` layout. Dispatch on actual ``ndim`` so both fixed-base multi-link articulations and single-body kinematic rigid objects are handled. ``_create_buffers`` similarly allocated the no-velocity fallback for ``_sim_bind_body_com_vel_w`` as 1D, which the ``derive_body_acceleration_from_body_com_velocities`` kernel rejected at launch time (expects 2D). Allocate it as ``(num_instances, 1)`` to match the kernel signature.
Add classmethod `NewtonManager.sanitize_world_state(env_ids)` that
zeroes per-world MJWarp solver scratch (qacc_warmstart, qfrc_*, cacc,
cfrc_*) and Newton State velocity/force buffers (joint_qd, body_qd,
body_f, body_qdd, body_parent_f) for the given worlds, then runs
eval_fk to re-derive body_q.
Generalizes the in-tree Factory NaN scrubber so any task on Newton can
recover NaN-divergent worlds in place via:
if has_nan_envs:
NewtonManager.sanitize_world_state(nan_env_ids)
Compared to mjwarp.reset_data: closes the gap on derived qfrc_* / cacc
/ cfrc_int / cfrc_ext that reset_data doesn't touch. cfrc_int in
particular is read-modify-written via wp.atomic_add in
mujoco_warp/_src/smooth.py:_cfrc_backward, so a stale NaN there
survives the next rne() call and re-contaminates qfrc_bias.
No-op for solvers without mjw_data (XPBD, Featherstone, Kamino) since
those don't exhibit the MuJoCo warm-start contamination pattern.
Without this, mid-loop cls._collision_pipeline.collide reads stale state.body_q from the previous solver step — accurate for most collision pairs, but for SDF/hydroelastic contact normals on fast-moving fingers it produces visibly wrong penetrations. Re-deriving body_q from joint_q before the re-collide adds one eval_fk per collision_decimation interval, which is cheap relative to the contact solve.
accessors) - Newton dep: newton[sim]@v1.2.0 (was @v1.2.0rc2 with hand-pinned mujoco / mujoco-warp; [sim] extras now supply both). - NewtonCfg.collision_substeps: re-collide every N substeps within a tick. Default 0 keeps current behavior. Used for nut_thread with num_substeps=8 / collision_substeps=2. - HydroelasticSDFCfg.moment_matching: PhysX patch-friction analog. - factory_env: Newton OSC path reads body_link_jacobian_w + mass_matrix from PR isaac-sim#5400 accessors instead of the v02 hand-rolled FK+eval_mass_matrix glue. Velocity stays Newton-specific (data adapter zeros fingertip vel on the zero-mass virtual body). - factory_control_newton: trimmed to just clamp_to_effort_limits — the rest (NewtonOSCBuffers, build_buffers, compute_arm_jacobian/mass_matrix, IK solver) was superseded by PR isaac-sim#5400. - factory_control: gate the null-space term on cfg.ctrl.disable_nullspace. - factory_env_cfg / factory_newton_setup: Newton preset (MJWarp, hydro SDF contacts), num_substeps=8, collision_substeps=2, sim.dt=1/120, decimation=8.
…ip_velocity_from_newton_state)
Selecting ``collision_cfg.sdf_hydroelastic_config=None`` on the Newton preset now drops Newton's hydroelastic pipeline and runs the SDF collision through pure penalty springs. Useful when the hydroelastic dependency is unavailable or for ablation studies. Tuning that changes per mode: - finger PD: hydroelastic relies on ``kh × penetration`` and runs with (1000, 10); SDF-only carries grip force through the PD spring and needs (2000, 40) to stop the nut slipping during threading. - nut/bolt material: hydroelastic uses ``ke=1e4`` / ``kd=100`` / ``gap=0``; SDF-only uses ``ke=1e7`` / ``kd=1e4`` / ``gap=5 mm`` so the penalty spring keeps the finger penetration inside the SDF narrow band. - finger shape flags: hydroelastic flips ``HYDROELASTIC`` + ``kh``; SDF-only writes ``ke`` / ``kd`` on the finger shapes (nut/bolt stay on Newton's default rigid pipeline). Mode is latched once in ``apply_cfg_overrides`` and read by the ``MODEL_INIT`` callback at builder time.
Adds a small ad-hoc knob to override the SDF-only finger PD without rebuilding the cfg surface. Reads ``FACTORY_SDF_FINGER_KP`` / ``FACTORY_SDF_FINGER_KD`` once in ``apply_cfg_overrides`` and applies them to the panda_hand actuator. Defaults match the values shipped in the SDF-only commit (kp=2000, kd=40).
Picks the success_rate peak from the FACTORY_SDF_FINGER_KP/KD sweep on Factory_develop_baseline.pth (64 envs / 450 steps): ( 2000, 40) → 0.500 (old default) ( 3000, 60) → 0.547 ( 5000, 100) → 0.516 ( 6000, 120) → 0.641 ( 8000, 160) → 0.656 <- peak (10000, 200) → 0.594 (12000, 240) → 0.594 Higher stiffness above 8 k starts pushing the finger SDF into the nut SDF instead of gripping; lower stiffness lets the nut slip. Both axes peak at 8 k / 160. Still well below the hydroelastic 0.95 — SDF-only remains an ablation, not a competitive mode.
…errides Tunes MJWarp's ``solver_cfg.iterations`` and ``solver_cfg.impratio`` at runtime via env vars so we can sweep them without rebuilding the cfg surface — matches the ``FACTORY_SDF_FINGER_KP/KD`` knob already on this file. Defaults of 0 keep the cfg-pinned values; positive values override. Motivation: SDF-only penalty contacts may need much higher solver iterations (50-200) and impratio (100-1000) than the defaults to keep stiff contact pairs converged during gripper-on-nut threading.
Empirical finding: keeping nut/bolt at Newton-example contact values (ke=1e7, kd=1e4) and bumping only the *finger* contact stiffness + damping by ~30× jumps SDF-only success on Factory_develop_baseline.pth from 0.625 to 0.891 (engage 0.656 -> 0.953). The default ``_KE_FINGER = 1e7`` was inherited from the Newton nut/bolt example which has no gripper, so the value was wrong for the finger pads under the gripper's saturated 40 N close force. Combined with stiffness sweep on finger (ke, kd): (1e7, 1e4) -> baseline (success 0.625) (3e7, 3e4) -> marginal (1e8, 1e5) -> better mean penetration, worse worst case (3e8, 3e5) -> best: success 0.891 <- new default (1e9, 1e6) -> solver oscillation, worst-case penetration grows Cranking solver iters (100) + impratio (1000) on top of the new defaults HURTS success_rate (0.766) — confirmed by the penetration probe (worst-case overlap grows from -0.72 mm to -2.66 mm).
Collapse multi-paragraph block comments / module docstrings to single short lines focused on the WHY when the WHAT is obvious from identifiers. Mirrors the style used in newton_visualizer / cfg classes elsewhere in the codebase. No behaviour change.
Artifact of the rebase: the eval_fk lines were preserved during the e42361a481 cherry-pick conflict resolution against an earlier prereqs that had them, but prereqs subsequently dropped them. Re-align.
On Newton, NaN-divergent worlds occasionally appear in the obs/state tensors (stiff SDF contacts under RL exploration). Detect via torch.isnan in _get_observations, mark the affected envs, substitute zeros so the rl_games policy head doesn't fail (std must be >= 0), and call NewtonManager.sanitize_world_state to scrub solver internals + Newton State buffers for those worlds before the next step. No-op on PhysX (self._is_newton is False) and on healthy Newton runs (self._nan_envs stays all-False). Mechanism lives in NewtonManager.sanitize_world_state on the prereqs branch; this commit just wires the call site.
PR isaac-sim#5729 renamed the knob. The comment-trim cherry-pick masked the edit by trimming the comment block that referenced `collision_substeps` by name, leaving the kwarg name itself stale. Caught by import-time AttributeError on FactoryEnvCfg construction.
…r moment_matching The Factory cherry-pick (e42361a481) was authored before PR isaac-sim#5228 landed and shipped its own moment_matching field at a different position with a different docstring. The conflict resolution kept Factory's variant and dropped PR isaac-sim#5228's, leaving a no-op diff against the prereqs branch (moves the field; rewrites the docstring; same value). Restore PR isaac-sim#5228's canonical position+docstring so prereqs and nut-thread-newton agree byte-for-byte on this file.
…decimate=1) Adds a third PresetCfg branch alongside 'newton' (hydroelastic) and 'physx': 'newton_sdf' uses SDF mesh contacts without hydroelastic, with num_substeps=10 and collision_decimation=1. Vanilla SDF's penalty-spring normal response is stiffer than hydroelastic distributed pressure. The denser substepping (0.83 ms substep dt) plus per-substep re-collide keeps contact normals fresh through the tick, which empirically avoids the contact-normal staleness that produces unstable threading under 8/2. Invoked as 'presets=newton_sdf' on the unified train/play scripts.
7 tasks
![isaaclab-review-bot[bot]](https://avatars.githubusercontent.com/u/157846462?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
🤖 Isaac Lab Review Bot
Summary
This PR adds Newton backend support to the Factory NutThread task, enabling both hydroelastic SDF contacts and vanilla penalty-spring SDF modes alongside the existing PhysX path. The implementation includes OSC control adaptation, NaN recovery mechanisms, and extensive contact tuning. Key concerns center around configuration complexity, global state management, and some potential edge cases in error handling.
Design Assessment
Architecture: Good separation — Newton-specific code is cleanly isolated into factory_control_newton.py and factory_newton_setup.py, keeping the shared PhysX path uncluttered. The PresetCfg pattern enables clean backend switching via CLI.
Newton/PhysX symmetry — The backend dispatch via _is_newton_backend() is straightforward. The conversion of ArticulationCfg → RigidObjectCfg for held/fixed assets follows the pattern in shadow_hand_vision.
Concerns:
- Global mutable state (
_use_hydroelastic) latched at import time could cause issues with multi-cfg scenarios - Monkey-patching the cloner (
_monkey_patch_cloner_no_simplify) is acknowledged as a workaround but should be tracked for upstream cleanup - The
sanitize_world_stateNaN recovery mechanism adds significant complexity to the simulation lifecycle
Findings
🔴 Critical: Unguarded division in mass-matrix inversion
factory_control.py:81-83 — The mass-matrix inversion torch.inverse(arm_mass_matrix) is called unconditionally. If the matrix becomes singular (which can happen under Newton's contact stack), this will produce NaN/Inf that the NaN-gate may not catch until the next step.
# Current:
arm_mass_matrix_inv = torch.inverse(arm_mass_matrix)
# Suggested: Add condition check
if torch.linalg.det(arm_mass_matrix).abs().min() < 1e-8:
logger.warning("Near-singular mass matrix detected")
# Return early or use pseudoinverse🟡 Warning: Broad exception handling absent in NaN recovery
newton_manager.py:1542-1580 — sanitize_world_state iterates over MJWarp fields and converts to torch, but if a field is unexpectedly shaped or typed, the conversion will silently fail. Consider adding validation:
# Current:
t = wp.to_torch(arr)
# Suggested:
try:
t = wp.to_torch(arr)
except Exception as e:
logger.warning(f"Failed to convert {field_name} for sanitization: {e}")
continue🟡 Warning: Global state mutation via _use_hydroelastic
factory_newton_setup.py:22-23 — This global is written in apply_cfg_overrides() and read in _model_init_callback(). If two Factory envs with different physics configs were instantiated sequentially (e.g., in tests), the second would inherit the first's latch:
# Latched in :func:`apply_cfg_overrides`; read by the MODEL_INIT callback.
_use_hydroelastic: bool = TrueConsider passing this through the callback closure or storing it on the cfg object itself.
🟡 Warning: Missing device consistency check in flag_nan_envs
newton_manager.py:1597-1608 — The mask tensor is ORed with _nan_env_mask_pending_reset, but there's no check that they're on the same device:
# Current:
cls._nan_env_mask_pending_reset |= mask
# Suggested: Ensure device match
if cls._nan_env_mask_pending_reset.device != mask.device:
mask = mask.to(cls._nan_env_mask_pending_reset.device)🔵 Suggestion: Consider making env-var overrides more discoverable
factory_newton_setup.py:53-72 — Multiple FACTORY_* env vars control runtime behavior but are only documented in comments. Consider adding a docstring listing them:
def apply_cfg_overrides(cfg: FactoryEnvCfg) -> None:
"""Apply Newton-only cfg overrides.
Environment variable overrides:
FACTORY_SDF_FINGER_KP/KD: Finger PD gains (SDF-only mode)
FACTORY_SOLVER_ITERATIONS: MJWarp solver iterations
FACTORY_SOLVER_IMPRATIO: MJWarp solver impratio
FACTORY_NUM_SUBSTEPS: Override num_substeps
FACTORY_NJMAX_MULT: Contact buffer multiplier
"""🔵 Suggestion: Early return in _compute_fingertip_velocity_from_newton_state
factory_env.py:181-226 — The error path resolves body indices each call via list comprehension and string matching. Cache the lookup or move to init:
# Current (called every step):
labels = list(model.body_label)
idxs = []
for env_idx in range(self.num_envs):
env_token = f"/env_{env_idx}/"
for i, lab in enumerate(labels):
if lab.endswith(...) and env_token in lab:
...
# Already done: The code does cache after first call via hasattr check ✓
# Good pattern.🔵 Suggestion: Collision filter pairs could be deduplicated
factory_newton_setup.py:280-291 — The nested loop appends pairs without checking for duplicates. While shape_collision_filter_pairs may handle this, explicit dedup would be safer:
# Suggested:
existing = set(builder.shape_collision_filter_pairs)
for base_i in base_shape_idxs:
for table_i in table_shape_idxs:
pair = (min(base_i, table_i), max(base_i, table_i))
if pair not in existing:
builder.shape_collision_filter_pairs.append(pair)
existing.add(pair)Test Coverage
Good coverage — The PR adds comprehensive tests:
test_newton_cfg_collision_decimation_warning: Parametrized warning-gate testtest_collision_decimation_invokes_mid_loop_collide: Integration test with collide-call countertest_sdf_config.py: 487 lines of SDF config unit tests covering_build_sdf_on_mesh,_apply_sdf_config,_create_sdf_collision_from_visual
Gap: No explicit test for the NaN recovery path (sanitize_world_state, flag_nan_envs, sanitize_pending_nan_envs). Consider adding a test that injects NaN into a world and verifies recovery.
CI Status
✅ labeler: pass (8s)
Verdict
Minor fixes needed — The PR is well-structured with good separation between backends and comprehensive testing. The main concerns are around global state management and edge-case error handling. These are not blocking for a draft PR but should be addressed before merge.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Factory NutThread support on the Newton backend
Description
Adds Newton-backend support to the Factory NutThread task (:class:
~isaaclab_tasks.direct.factory.FactoryEnv). The existing PhysX path remains the default and is behaviour-preserving (see Type of change) — switching backends is a one-line CLI swap:The Newton hydroelastic preset trained to 100% success on the 200-epoch benchmark (PhysX reached 99.2% in the same run); the SDF preset trains stably with the deferred-NaN-recovery design but its contact-stack tuning is still in progress (see SDF preset is still being tuned).
This is draft — it depends on #5835 — Newton prereqs for Factory NutThread support (see Dependencies).
Rollout video
Reward and success are running (accumulating live as the rollout plays).
nut-thread-physx_vs_newton-hydro_vs_newton-sdf_h264.mp4
Dependencies
This PR sits on top of #5835 — Newton prereqs for Factory NutThread support. That branch bundles four Newton-side prerequisites; the Factory changes here only become functional once those land. The prereqs PR is also draft and is itself a bundle of:
NewtonCfg.collision_decimation(@mzamoramora-nvidia)RigidObjectDatashape fix for kinematic single-body assetsNewtonManager.sanitize_world_statehook + deferred NaN-reset queueWhy the prereqs commits also appear here (temporary)
This PR's branch temporarily carries the prereqs commits so GitHub has something to render against
develop. They are not part of this PR's review surface — they belong to the prereqs PR — and will be dropped via rebase once the prereqs PR (or its upstream pieces) land.Compare view (Factory-only diff, prereqs excluded):
mzamoramora-nvidia/IsaacLab@nut-thread-newton-prereqs...nut-thread-newton
What's in this PR (Factory-only diff)
direct/factory/factory_env.pydirect/factory/factory_env_cfg.pynewtonandnewton_sdfpresets toFactoryPhysicsCfg(extendsPresetCfg).direct/factory/factory_control.pydirect/factory/factory_control_newton.py(new)newton.selection.ArticulationViewto the J/M shapes the Factory OSC math expects, foldsjoint_armatureinto the mass-matrix diagonal.direct/factory/factory_newton_setup.py(new)The two new files (
factory_control_newton.py,factory_newton_setup.py) keep the Newton-only branches out of the sharedfactory_env.py/factory_control.pyso the PhysX path stays uncluttered and the Newton path stays inspectable in one place.Newton preset details
The vanilla-SDF preset uses more substeps (10 vs 8) and a tighter
collision_decimation(1 vs 2) — empirically these values are needed to keep the SDF run stable on this task. Both rely onNewtonCfg.collision_decimationfrom PR #5729.SDF preset is still being tuned
The
newton_sdfpreset is the most contact-sensitive of the three backends and the contact-stack tuning here is not final:(kp=8000, kd=160)(seefactory_newton_setup.py). Softer settings also look promising in contact-sweep tests — e.g.(kp=1000, kd=10)and the even-softer(kp=500, kd=5). The trade-off is grip stiffness vs penetration depth, and the right operating point depends on the contact-springke/kdchosen alongside.ke/kd, and the bolt SDF resolution is open work.collision_decimationis a perf knob:cd=1is the safest setting but pays for an extra collide-pass every substep. Raisingcdrecovers fps but eventually destabilises contacts; the right value should be tuned per task.factory_newton_setup.pyexposesFACTORY_SDF_FINGER_KP/KD/FACTORY_NUM_SUBSTEPS/FACTORY_SOLVER_ITERATIONSenv-vars so sweeps can be run without code changes.These are all knobs in
factory_newton_setup.py— tuning continues but is out of scope for this PR. The values shipped here are a working baseline that lets the SDF preset learn end-to-end.NaN-recovery via the deferred-sanitize queue
FactoryEnv._get_observationsflags any world that produced a NaN this step:_get_rewardsapplies the samenan_to_numso PPO never sees a NaN reward. The actual MJWarp buffer scrub is deferred to_reset_idx:This deliberately leaves live warm-starts intact between divergence and episode boundary — earlier eager mid-rollout scrubs regressed hydroelastic training, and deferring the scrub recovered it. Under this design the Newton-SDF preset trains end-to-end where earlier attempts produced
rew=nanfrom epoch 1 (see Training results).Training results
200-epoch rl_games PPO runs on
Isaac-Factory-NutThread-Direct-v0, 128 envs, seed 0, default rl_games config (save_best_after=10,save_frequency=100). Hardware: single NVIDIA L4 (24 GB).Observations from this single-seed run:
rew=nanfrom epoch 1; the deferred-sanitize design changed during this work and the SDF preset stopped diverging. At the latest sample (epoch 175 of 200), success was 98.4% — within ~1.6 percentage points of hydro's 100% — and reward was still climbing slowly. On a per-epoch basis the SDF preset is the slowest learner of the three (unsurprising given the stiffer penalty-spring contact response), and the contact-stack tuning is still in progress (see SDF preset is still being tuned).These are single-seed observations on one task and should be read as "the Newton backend works end-to-end on Factory NutThread", not as a benchmark.
Related branches
End-to-end Factory + Newton tooling (eval, rollout video,
VIDEO_HOWTO.md) lives on the follow-up branchnut-thread-newton-bundle-full.Compare view (bundle-full's diff vs this PR):
mzamoramora-nvidia/IsaacLab@nut-thread-newton...nut-thread-newton-bundle-full
Type of change
PresetCfgwrappers, but the PhysX branches remain functionally equivalent todevelop(verified by matching the 200-epoch PhysX baseline reward trajectory, see Test plan).Test plan
./isaaclab.sh -fIsaac-Factory-NutThread-Direct-v0The two Newton trainings are the primary functional gate — they exercise the full backend dispatch (
factory_control_newton,factory_newton_setup, preset selection, NaN gate, contact pipeline). No dedicated unit tests are added in this PR; the task-level training runs are the integration test, and the unit-test surface lives in the prereqs PR (PR #5228 / PR #5729 ship 53 parametrized tests between them).Checklist
pre-commitchecks with./isaaclab.sh --formatPresetCfgdocstring covers both Newton presets)source/<pkg>/changelog.d/for every touched package (source/isaaclab_tasks/changelog.d/nut-thread-newton.minor.rst,source/isaaclab_newton/changelog.d/nut-thread-newton.rst)CONTRIBUTORS.mdor my name already exists there