Skip to content

feat: Add a stateful UFM simulator with standalone and machine-a-tron-hosted modes #3528

Description

@poroh

Is this a new feature, an enhancement, or a change to existing functionality?

New Feature

How would you describe the priority of this feature request

Medium

Please provide a clear description of problem this feature solves

NICo integration tests currently cannot exercise the real UFM REST client without access to a physical InfiniBand fabric. The existing in-memory IB mock bypasses HTTP and does not validate UFM API compatibility.

A separate UFM simulator service supports large and multi-source fabrics, but requiring another process makes small machine-a-tron deployments unnecessarily complicated.

Feature Description

Implement a reusable, stateful UFM simulator component that can run in either of these modes:

  • As a standalone binary representing a simulated InfiniBand fabric.
  • Embedded in machine-a-tron and enabled through configuration for simple, single-process deployments.

Both modes must expose the same UFM-compatible REST API and use the same simulator implementation.

Describe your ideal solution

Create a reusable UFM simulator library containing:

  • The UFM-compatible HTTP router.
  • Port and partition state.
  • Partition membership and QoS behavior.
  • Machine inventory reconciliation.
  • Fault and latency simulation hooks.

Provide a standalone binary around this library. It can reconcile inventory from multiple machine-a-tron /machines/status endpoints.

Allow machine-a-tron to optionally host the same router:

[ufm_simulator]
enabled = true
fabric_name = "simulated"
include_local_inventory = true

When include_local_inventory is enabled, the embedded simulator consumes the same underlying inventory used to generate /machines/status. It must not maintain a second copy of machine or GUID definitions.

The embedded mode may also accept additional machine-a-tron status URLs, allowing one machine-a-tron process to host a UFM simulator for inventory aggregated from several machine-a-tron instances.

Initially support the UFM operations used by NICo’s RestIBFabric client:

  • Retrieve the UFM version and subnet-manager configuration.
  • List compute ports.
  • List, retrieve, create, and delete partition memberships.
  • Remove GUIDs from a partition.
  • Update partition QoS configuration.

State and API behavior must be identical in standalone and embedded modes. Contract tests should run the existing RestIBFabric client against both launch modes.

Describe any alternatives you have considered

Only provide a standalone binary

This provides clean service boundaries and independent scaling, but forces every small machine-a-tron environment to deploy and configure another service.

Only embed UFM in machine-a-tron

This simplifies local deployment but couples UFM capacity and lifecycle to machine-a-tron. It also makes aggregation across many machine-a-tron instances harder to operate.

Implement separate embedded and standalone simulators

This would allow each mode to evolve independently but would duplicate UFM behavior and create compatibility drift. Both modes should wrap the same library and router.

Additional context

No response

Code of Conduct

  • I agree to follow NVIDIA Infra Controller's Code of Conduct
  • I have searched the open feature requests and have found no duplicates for this feature request

Metadata

Metadata

Assignees

No one assigned

    Labels

    internalinternal work transparent to end-userscaling

    Fields

    No fields configured for Enhancement.

    Projects

    Status
    Triage

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions