feat: support scalable UFM simulation from machine-a-tron inventory

[poroh]

Is this a new feature, an enhancement, or a change to existing functionality?

Enhancement

How would you describe the priority of this feature request

Medium

Please provide a clear description of problem this feature solves

Machine-a-tron can advertise InfiniBand hardware through machine discovery, but there is no UFM service that discovers and manages those simulated ports. Machine-a-tron integration tests consequently run with InfiniBand disabled, while existing IB tests use an in-memory fabric implementation that bypasses the production UFM REST client.

We need a shared UFM simulator that:

• Discovers ports from one or more configured machine-a-tron instances.
• Continuously reconciles their inventory.
• Exposes the UFM REST contract used by NICo.
• Can run as a standalone service or optionally be hosted by machine-a-tron for simple deployments.
• Coexists with a real UFM as a separate named fabric.

Requiring one UFM simulator per machine-a-tron process would couple fabric capacity to the capacity of a single machine simulator and make it difficult to represent one large fabric assembled from several machine-a-tron instances. Conversely, requiring a separate process for every small test environment would make simple deployments unnecessarily complex. The same simulator implementation should therefore support both standalone and machine-a-tron-hosted launch modes.

NICo contains partial multi-fabric support, but startup currently permits only one fabric and several partition, credential, and cleanup paths still hard-code the default fabric.

Feature Description

When machine-a-tron exposes IB-capable machines, a reusable UFM simulator shall discover and reconcile their IB inventory into one simulated fabric.

The simulator shall be runnable as a standalone binary or optionally hosted by machine-a-tron. Both modes shall use the same implementation and expose the same UFM REST v3 behavior.

The simulator shall support inventory from multiple machine-a-tron instances, allowing their machine-backed ports to be combined into one simulated fabric.

When both a real UFM and the simulator are configured, NICo shall treat them as independent named fabrics and route all partition operations to the owning fabric.

Describe your ideal solution

Machine-a-tron inventory

Extend the existing stable GET /machines/status response with the InfiniBand inventory required by the UFM simulator. Do not introduce a second machine inventory endpoint or a separate control listener.

Preserve the existing response fields and behavior for the machine-a-tron Web UI. Add:

• A stable response-level source_id identifying the machine-a-tron deployment.
• A monotonically increasing inventory generation that changes when machines, IB ports, GUIDs, or simulated port states change.
• An infiniband_ports collection on each applicable machine containing its port GUIDs and simulated states.

Reuse the existing fields:

• mat_id as the machine-a-tron machine identifier.
• machine_id as the optional NICo machine identifier learned after discovery.
• hardware_type as the simulated hardware type.

InfiniBand port definitions must become part of the machine's static inventory. Redfish discovery and /machines/status must consume the same definitions so they cannot report different GUIDs for the same machine. GUIDs must be deterministic, globally unique across machine-a-tron instances, and stable across restarts.

The response remains a complete inventory snapshot so a consumer can determine when machines or ports have disappeared.

UFM simulator

Add a reusable, stateful UFM simulator component representing one simulated fabric. Provide two launch modes around the same implementation:

• A standalone binary for independent scaling and multi-source aggregation.
• Optional hosting by machine-a-tron for simple, single-process deployments.

Both modes shall use the same HTTP router, state model, inventory reconciliation, and contract tests.

The simulator shall expose the UFM REST v3 operations currently used by carbide-ib-fabric, including:

• UFM version and subnet-manager configuration.
• Compute-port listing.
• Partition listing, lookup, and creation.
• GUID-to-pkey binding and unbinding.
• Partition QoS updates.

The simulator shall maintain port, partition, membership, and QoS state independently from machine-a-tron's machine actors. It shall expose metrics for request latency, response size, port count, partition count, and reconciliation health.

When hosted by machine-a-tron, the simulator shall be optional and independently configurable. It shall consume the same underlying local inventory used by /machines/status, without maintaining a second copy of machine or GUID definitions. The hosted mode may also reconcile additional configured machine-a-tron sources.

Disabling the hosted simulator must leave existing machine-a-tron behavior unchanged.

Inventory discovery and reconciliation

Configure the simulator with zero or more remote machine-a-tron /machines/status URLs. A standalone simulator normally uses one or more remote sources. A machine-a-tron-hosted simulator may combine its local inventory with remote sources.

For every source, the simulator shall:

1. Poll complete inventory snapshots concurrently.
2. Track inventory ownership by stable source_id.
3. Track (epoch_id, generation) per source, reset the generation watermark when the epoch changes, and ignore duplicate or older generations only within the same epoch.
4. Add newly discovered GUIDs.
5. Update changed port states.
6. Remove GUIDs that disappear from a successful complete snapshot.
7. Preserve pkey memberships for unchanged GUIDs.
8. Detect and report GUID conflicts between sources.
9. Retain the last successful inventory across temporary source failures.
10. Mark ports down or remove them after a configurable failure grace period.

The simulator must be able to restart and reconstruct its machine-backed port inventory without requiring machine-a-tron to push registrations.

Real and simulated fabric coexistence

Complete NICo's named-fabric support so a real UFM and a simulated UFM can be configured together as independent fabrics.

This requires:

• Associating IB partitions with a fabric_id.
• Scoping pkey allocation and uniqueness by fabric.
• Managing UFM credentials per fabric.
• Routing partition reconciliation, QoS, binding, release, deletion, and administrative cleanup to the owning fabric.
• Rejecting partition and machine-GUID fabric mismatches.
• Removing the single-fabric startup restriction.

The simulator must never proxy or forward requests to the real UFM.

Describe any alternatives you have considered

Only provide a standalone UFM simulator

This provides clean service boundaries and independent scaling, but it forces every small machine-a-tron environment to deploy and configure another process.

Only embed UFM in machine-a-tron

This simplifies small deployments, but making it the only launch mode couples the simulator's lifecycle and capacity to one machine-a-tron process. Providing both standalone and hosted modes preserves simple deployment while allowing independent scaling and aggregation.

Implement separate standalone and embedded simulators

Separate implementations would duplicate UFM behavior and allow their API contracts to drift. Both launch modes should wrap the same reusable simulator component.

Add a separate InfiniBand inventory endpoint

A dedicated endpoint would duplicate machine identity, hardware type, and lifecycle data already exposed by /machines/status. Extending the existing stable response additively provides one authoritative machine inventory contract and preserves its existing Web use.

Discover ports by querying simulated Redfish endpoints

The UFM simulator would need to understand hardware-specific discovery representations, contact every simulated BMC, and reconstruct machine ownership. It would also couple UFM simulation scale to the number of running BMC simulations.

Push registrations from machine-a-tron

Push registration requires lifecycle coordination and cleanup when either process restarts. Pulling complete snapshots allows the UFM simulator to reconstruct and reconcile its state.

Hybrid proxy in front of real UFM

A proxy would sit in the mutation path to real hardware and would need to merge reads and safely split binding, unbinding, partition, and QoS writes. Separate named fabrics provide stronger isolation.

Use only the in-memory MockIBFabric

The in-memory implementation remains useful for component tests, but it does not exercise authentication, HTTP behavior, serialization, response sizes, or the production UFM REST client.

Additional context

No response

Code of Conduct

• I agree to follow NVIDIA Infra Controller's Code of Conduct
• I have searched the open feature requests and have found no duplicates for this feature request