diff --git a/docs/provisioning/boot-interfaces-and-dpu-modes.md b/docs/provisioning/boot-interfaces-and-dpu-modes.md index c1362d3c32..c6a6e6239b 100644 --- a/docs/provisioning/boot-interfaces-and-dpu-modes.md +++ b/docs/provisioning/boot-interfaces-and-dpu-modes.md @@ -1,6 +1,6 @@ # Boot Interfaces and DPU Modes -This guide explains how NICo decides **which interface a host boots from**, how a host's **DPUs are managed**, and how operators configure both through the Expected Machines table. It is the deep companion to [Ingesting Hosts](ingesting-hosts.md): that page covers the end-to-end ingest flow and the basic `expected_machines.json`; this page covers the per-host and per-NIC knobs (`dpu_mode`, `host_nics`), **what the defaults do when you set nothing**, and how a boot device is chosen and applied behind the scenes. +This guide explains how NICo decides **which interface a host boots from**, how a host's **DPUs are managed**, and how operators configure both through the Expected Machines table. It is the deep companion to [Ingesting Hosts](ingesting-hosts.md), which covers the end-to-end ingest flow and the basic `expected_machines.json`; this page covers the per-host and per-NIC knobs (`dpu_mode`, `host_nics`), **what the defaults do when you set nothing**, and how a boot device is chosen and applied behind the scenes. For the DHCP and network-segment substrate these knobs sit on (how a relay's `giaddr` maps to a segment), see [IP and Network Configuration](ip-and-network-configuration.md). @@ -15,9 +15,9 @@ Historically, two separate decisions were conflated into "what kind of host is t | Axis | Question | Controlled by | |---|---|---| | **DPU management** | Does NICo manage this host's DPUs (upgrade them, run agents, serve the host's admin network over the DPU overlay)? | `dpu_mode` | -| **Boot interface** | Which NIC does the host OS actually boot from and run its management network on? | the host's **primary** interface | +| **Boot interface** | Which NIC does the host OS boot from and run its management network on? | the host's **primary** interface | -A "normal" host couples them (managed DPU + boot through that DPU). But they are independent: you can keep a host's DPUs **managed** and still boot it from a plain **integrated NIC**. This guide treats the two axes separately, because the configuration knobs are separate. +A "normal" host couples them (a managed DPU it also boots through). But they are independent: you can keep a host's DPUs **managed** and still boot it from a plain **integrated NIC**. This guide treats the two axes separately, because the configuration knobs are separate. ### Network segment types @@ -36,6 +36,8 @@ A host's management network lives on one of a few segment types. Which one depen In NICo the **boot interface and the primary interface are the same thing by construction**: a host's `primary` interface is the one its boot order targets. A boot interface is a `(MAC, Redfish interface id)` pair — the MAC identifies the NIC on the wire; the Redfish id lets NICo set the boot order on the BMC. NICo refers to this pair as the host's `MachineBootInterface`. +NICo keeps both halves because the MAC alone can go stale: a NIC can drop out of the BMC's Redfish inventory (taking its MAC with it) while the interface id remains addressable. When NICo targets a boot operation at the BMC, it tries the MAC first and, when it knows the full pair, retries with the interface id on any error — so boot configuration keeps working when the MAC has vanished from the inventory, as happens after a DPU-to-NIC-mode flip or when a NIC de-enumerates across a reboot. + --- ## 2. Configuring via Expected Machines and the defaults @@ -50,7 +52,7 @@ Boot and DPU configuration is **declarative**: you describe the host in the Expe - Site-explorer **auto-selects the boot interface**: the lowest-PCI DPU host-PF (the NIC a DPU presents to the host). - The host's IP comes from whichever segment its DHCP relay lands in (see [IP and Network Configuration](ip-and-network-configuration.md)). -So a standard DPU host is handled entirely by defaults. The knobs below exist for the hosts that *don't* fit that mold. +So a standard DPU host is handled entirely by defaults; the knobs below exist for the exceptions. ### `dpu_mode` @@ -60,7 +62,7 @@ So a standard DPU host is handled entirely by defaults. The knobs below exist fo | `nic_mode` / `nic-mode` | DPU hardware is present but treated as a **plain NIC**. Site-explorer explores it but does **not** link or manage it; the host boots on **HostInband**. | | `no_dpu` / `no-dpu` | No DPU hardware at all — a plain host NIC on **HostInband**. DPU exploration is skipped entirely. | -**Resolution order:** a per-host `dpu_mode` on the Expected Machine wins; if unset, the site-wide `[site_explorer] dpu_mode` setting applies; if that too is unset, the default is `dpu_mode`. +**Resolution order:** a per-host `dpu_mode` on the Expected Machine wins; if unset, the site-wide `[site_explorer] dpu_mode` setting applies; if it too is unset, the default is `dpu_mode`. ### `host_nics` (per-NIC declaration) @@ -74,7 +76,7 @@ The optional `host_nics` array declares specifics for individual host NICs. Each | `fixed_ip` / `fixed_mask` / `fixed_gateway` | string | Static IP assignment for the NIC, pre-allocated at upload time. | Dynamic allocation. | | `nic_type` | string (legacy) | A free-form segment hint, **superseded by `network_segment_type`**. Kept for backward compatibility only. | — | -> **What `network_segment_type` actually does.** A NIC's segment is normally determined by its DHCP relay: NICo picks the segment whose prefix *contains* the relay address. Where segment prefixes **nest or overlap** — for example a `/27` HostInband segment inside a `/24` underlay — one relay matches several segments. `network_segment_type` narrows that to the segment of the named type. If a relay maps unambiguously to one segment (the common case), this field is unnecessary. +> **What `network_segment_type` does.** A NIC's segment is normally determined by its DHCP relay: NICo picks the segment whose prefix *contains* the relay address. Where segment prefixes **nest or overlap** — for example a `/27` HostInband segment inside a `/24` underlay — one relay matches several segments. `network_segment_type` narrows the match to the segment of the named type. If a relay maps unambiguously to one segment (the common case), this field is unnecessary. **JSON** (an Expected Machine entry): @@ -131,7 +133,7 @@ A plain server with one or more host NICs and no DPU. Declare `no_dpu` and mark } ``` -The host boots from that NIC on HostInband and gets its IP from central NICo DHCP. +The host boots from the declared NIC on HostInband and gets its IP from central NICo DHCP. ### 3.3 DPU in NIC mode @@ -165,7 +167,7 @@ NICo keeps the DPUs explored, linked, and underlay-addressed (running agents for ### 3.5 Flipping a DPU to NIC mode -To change a host that's already ingested (e.g. from managed-DPU to NIC mode), update its Expected Machine `dpu_mode`, then force-delete and let it re-ingest so site-explorer re-explores and applies the new mode: +To change an already-ingested host (e.g. from managed-DPU to NIC mode), update its Expected Machine `dpu_mode`, then force-delete and let it re-ingest so site-explorer re-explores and applies the new mode: ```bash nico-admin-cli -a em patch --bmc-mac-address --dpu-mode nic-mode @@ -196,6 +198,7 @@ All of these are **admin-only**; the Forge gRPC service enforces admin authoriza | admin-cli | Forge RPC | Purpose | |---|---|---| +| `machine boot-interfaces ` | `GetMachineBootInterfaces` | Show the boot interface as recorded in every store ([Section 7](#7-the-boot-interface-data-model)), the effective selection, and a divergence flag ([Section 8](#8-verifying-and-troubleshooting)). Read-only. | | `managed-host set-primary-interface [--reboot]` | `SetPrimaryInterface` | Designate a machine interface as the host's primary/boot interface. **The modern form.** | | `managed-host set-primary-dpu [--reboot]` | `SetPrimaryDpu` | Designate a DPU as primary. *Deprecated — prefer `set-primary-interface`.* | | `boot-override set [--custom-pxe ] [--custom-user-data ]` | `SetMachineBootOverride` | Override the iPXE script / cloud-init user-data served at boot. | @@ -204,6 +207,8 @@ All of these are **admin-only**; the Forge gRPC service enforces admin authoriza > Setting the DPU-first boot order directly (by MAC) is also exposed as a one-off action through the web UI ([Section 5](#5-web-ui)). Under normal operation the machine-controller sets the boot order automatically during ingestion ([Section 6](#6-behind-the-scenes-how-a-boot-device-is-chosen-and-set)). +> **The direct-to-BMC escape hatch.** The `redfish` command group (`redfish machine-setup`, `redfish set-boot-order-dpu-first`, `redfish is-boot-order-setup`, …) talks to a BMC directly, using operator-supplied credentials and a hand-typed `--boot-interface-mac`. It consults none of the stores in this guide and gets no interface-id fallback. It is useful against a BMC outside NICo's management; for managed hosts, prefer the commands above so the database and the BMC stay in agreement. + ### Ingestion control | admin-cli | Forge RPC | Purpose | @@ -228,7 +233,7 @@ The NICo admin web UI (`/admin/…`) is primarily for **visibility**, with a foc **Act:** - **Set DPU First Boot Order** (by MAC) — on a machine or explored endpoint. -- **Restore Boot Interface** — one-click re-apply of the host's resolved boot interface. It uses the machine's designated primary once managed, or site-explorer's automatic default before that. Handy right after a DPU↔NIC-mode flip. +- **Restore Boot Interface** — one-click re-apply of the host's resolved boot interface. It uses the machine's designated primary once managed, or site-explorer's automatic default beforehand. Handy right after a DPU↔NIC-mode flip. - **Machine Setup** — prepare an endpoint for ingestion (optionally with a boot-interface MAC; Dell endpoints require it). - Endpoint controls — **Re-Explore**, **Refresh**, **Clear Last Error**, **Pause/Resume Remediation**, plus power, Secure Boot, lockdown, and BMC-reset actions. @@ -264,26 +269,50 @@ At each boot-config step the controller resolves the target via `load_boot_predi 2. Otherwise, the **boot prediction** (`pick_boot_prediction`) — used *before* the first lease. 3. Otherwise, a classification: - **AwaitingNic** — a zero-DPU/NIC-mode host whose boot NIC hasn't appeared yet; wait. - - **Missing** — a host that *should* have a boot interface (it has DPUs) but doesn't; a fault to investigate. + - **Missing** — a DPU host *should* have a boot interface but doesn't; a fault to investigate. -> **Key timing.** A host has no `machine_interfaces` row until its first DHCP lease. Predictions are what let the controller configure boot **before** that lease. Once the host leases and the prediction is promoted to an owned row, the **owned row supersedes** the prediction. +> **Key timing.** A host has no `machine_interfaces` row until its first DHCP lease. Predictions let the controller configure boot **before** the first lease. Once the host leases and the prediction is promoted to an owned row, the **owned row supersedes** the prediction. ### Applying the boot order -- `configure_host_bios` (at `WaitingForPlatformConfiguration`) calls Redfish `machine_setup` with the resolved boot interface; on Dell this schedules a BIOS job (`WaitingForBiosJob`). +- `configure_host_bios` (at `WaitingForPlatformConfiguration`) calls Redfish `machine_setup` with the resolved boot interface. This configures the BIOS, including the **UEFI HTTP-boot device** pointed at the boot NIC; on Dell it schedules a BIOS job (`WaitingForBiosJob`). - `PollingBiosSetup` verifies the BIOS settings took. -- `SetBootOrder` sets the host boot order via Redfish — **DPU-first** for DPU hosts; for zero-DPU/NIC-mode hosts it targets the resolved HostInband interface (a "no DPU" response from the BMC is expected and treated as success). +- `SetBootOrder` is **check-first**: it verifies the HTTP-boot device and the boot order before writing anything, and a host whose configuration is already in place skips straight to verification. Nothing is re-applied unless something drifted — two BIOS writes never share one pass (on Dell they would collide in a single pending configuration job). +- If the check finds the HTTP-boot device **reverted** — the boot NIC dropped out of the BMC's Redfish inventory across a reboot, which some NICs do — `SetBootOrder` re-asserts it with `machine_setup` and reboots the host to apply the change first. A `WaitForHttpBootDeviceApplied` substate then polls for up to 10 minutes; each expired window spends one retry from a budget of 3 and returns for a fresh re-assert, and once the budget is exhausted the host is surfaced for manual intervention ([Section 8](#8-verifying-and-troubleshooting)). +- With the HTTP-boot device in place, the boot order itself is set via Redfish — **DPU-first** for DPU hosts; for zero-DPU/NIC-mode hosts it targets the resolved HostInband interface (a "no DPU" response from the BMC is expected and treated as success). - On a reprovision repair, `check_host_boot_config` re-checks BIOS + boot order and only remediates if they drifted. --- ## 7. The boot-interface data model -The boot interface flows through three tables: **predicted → managed → retained**. +A host's boot interface is recorded in **four stores**. Three form the machine's own lineage — **predicted → managed → retained** — and the fourth is the **explored default** site-explorer keeps per BMC endpoint, for hosts no machine owns yet ([Section 7.4](#74-selection-precedence)). `nico-admin-cli machine boot-interfaces` prints all four side by side ([Section 8](#8-verifying-and-troubleshooting)). + +```mermaid +flowchart LR + EXPM["expected_machines.host_nics
declared intent — input, not one of the four stores"] + subgraph lineage["The machine's own lineage"] + PRED[("predicted_machine_interfaces")] + OWNED[("machine_interfaces
authoritative once owned")] + RET[("retained_boot_interfaces")] + end + EXPL[("explored_endpoints
automatic default per BMC endpoint")] + ACT["boot actions
machine_setup, boot order"] + + EXPM -->|"machine creation mints predictions"| PRED + PRED -->|"first DHCP lease promotes"| OWNED + OWNED -->|"deletion retains the Redfish id"| RET + RET -->|"the next row for the MAC consumes"| OWNED + OWNED -->|"owned machines"| ACT + PRED -->|"before the first lease"| ACT + EXPL -->|"endpoints no machine owns"| ACT +``` + +Every exploration pass refreshes the Redfish interface id on owned rows, predictions, and the explored default alike, so all three stay as current as the last healthy exploration. ### 7.1 Predicted (`predicted_machine_interfaces`) -Site-explorer mints a prediction per declared host NIC **before** the host's first DHCP lease. A prediction carries `machine_id`, `mac_address`, `network_segment_type`, the operator's declared `primary` intent, and the `boot_interface_id` (the Redfish `EthernetInterface.Id`, captured from the exploration report once available). Predictions are what the controller uses to configure boot pre-lease. +Site-explorer mints a prediction per declared host NIC **before** the host's first DHCP lease. A prediction records `machine_id`, `mac_address`, `network_segment_type`, the operator's declared `primary` intent, and the `boot_interface_id` (the Redfish `EthernetInterface.Id`, captured from the exploration report once available). The controller uses predictions to configure boot pre-lease. ### 7.2 Managed (`machine_interfaces`) — promotion @@ -301,9 +330,9 @@ The owned table is **Store B**: the authoritative source of truth once a host is The Redfish boot-interface id is the one fact a MAC cannot always rediscover after deletion (a DPU/NIC-mode flip can drop the MAC from BMC reports while the id stays stable; a re-ingested host needs to be targeted for boot before a fresh exploration). So: - On **deletion** of a `machine_interfaces` row, its `boot_interface_id` is **upserted** into `retained_boot_interfaces` (keyed by MAC; newest wins). -- On **creation** of a new row, any retained id for that MAC is **consumed** and applied — provided it's within the configured `retained_boot_interface_window` (default: no expiry, i.e. retained forever; set a window to bound recycled-MAC reuse). +- On **creation** of a new row, any retained id for the new row's MAC is **consumed** and applied — provided it's within the configured `retained_boot_interface_window` (default: no expiry, i.e. retained forever; set a window to bound recycled-MAC reuse). -This is what carries a host's boot target across a force-delete / re-ingest gap. +This preserves a host's boot target across a force-delete / re-ingest gap. ### 7.4 Selection precedence @@ -321,7 +350,15 @@ The same precedence applies wherever NICo picks a boot interface, over owned row ## 8. Verifying and troubleshooting -**Check a host's boot interface:** +**First stop — the four-store view:** + +```bash +nico-admin-cli -a machine boot-interfaces +``` + +This read-only command prints the machine's boot interface as recorded in all four stores ([Section 7](#7-the-boot-interface-data-model)) side by side — owned rows, predictions, the explored default, and retained pairs (stale ones included) — plus the **effective** boot interface NICo would select, and a `divergent` flag set when the boot-selection signals — the effective owned selection, the explored default, and any primary-flagged prediction — name more than one distinct boot MAC (retained records and non-primary rows don't count toward disagreement). Divergence is normal mid-transition, for example between a mode flip and the re-ingest; persistent divergence on a settled host is worth investigating. For scripting, the global `--output json|yaml` flag applies. + +**For a quick primary check:** ```bash nico-admin-cli -a managed-host show @@ -335,6 +372,7 @@ The interfaces section shows each NIC's MAC, segment, and which one is `primary` |---|---| | `boot_interface_mac_mismatch` (pairing blocker) | The host's boot MAC doesn't match any discovered DPU's pf0 MAC. Expected for an integrated-NIC host — declare the integrated NIC `primary` (see [3.4](#34-boot-an-integrated-nic-while-keeping-the-dpus-managed)); otherwise check the exploration reports. See [Ingesting Hosts → pairing blockers](ingesting-hosts.md#common-blockers-during-host--dpu-pairing). | | Host stuck waiting for a boot NIC | A zero-DPU/NIC-mode host whose boot NIC hasn't leased yet (`AwaitingNic`). Confirm the NIC is cabled and DHCP-reachable on its HostInband segment. | +| `HTTP boot device on host … still not applied after 3 re-asserts` | The boot NIC keeps dropping out of the BMC's Redfish inventory across reboots, so the re-asserted HTTP-boot device never verifies and the retry budget runs out ([Section 6](#6-behind-the-scenes-how-a-boot-device-is-chosen-and-set)). Investigate why the NIC doesn't enumerate (NIC/BMC firmware); once it appears in the BMC's `EthernetInterfaces`, use **Machine Setup** in the web UI to re-assert and reboot. See the [Stuck Objects playbook](../playbooks/stuck_objects/stuck_objects.md) for restarting a failed state machine. | | Boot interface wrong after a DPU↔NIC-mode flip | Use **Restore Boot Interface** in the web UI, or re-ingest ([3.5](#35-flipping-a-dpu-to-nic-mode)). | | DPU mode "unknown" (`dpu_nic_mode_unknown`) | DPU BMC firmware too old to report mode. Install a fresh DPU OS — see [Ingesting Hosts](ingesting-hosts.md#dpu-related-issues-installing-a-fresh-dpu-os). | diff --git a/docs/provisioning/ingesting-hosts.md b/docs/provisioning/ingesting-hosts.md index 0a7b0b69f1..1b6dda4150 100644 --- a/docs/provisioning/ingesting-hosts.md +++ b/docs/provisioning/ingesting-hosts.md @@ -189,7 +189,8 @@ Each entry supports additional optional fields: ``` - **`dpf_enabled`** (bool): Enable/disable DPF for this host. -- **`dpu_mode`** (`"dpu_mode"` | `"nic_mode"` | `"no_dpu"`): Per-host DPU operating mode. +- **`dpu_mode`** (`"dpu_mode"` | `"nic_mode"` | `"no_dpu"`): Per-host DPU operating mode — see [Boot Interfaces and DPU Modes](boot-interfaces-and-dpu-modes.md). +- **`host_nics`** (array): Per-NIC declarations — the boot/primary NIC, its network segment type, and static IPs — see [Boot Interfaces and DPU Modes](boot-interfaces-and-dpu-modes.md). - **`bmc_retain_credentials`** (bool): Skip BMC password rotation. - **`default_pause_ingestion_and_poweron`** (bool): Pause ingestion and power-on for this host. - **`bmc_ip_address`** (string): Static BMC IP (pre-allocates a machine interface).