docs: refresh the boot-interface operator guide

docs/provisioning/boot-interfaces-and-dpu-modes.md

@@ -1,6 +1,6 @@
 # Boot Interfaces and DPU Modes
 
-This guide explains how NICo decides **which interface a host boots from**, how a host's **DPUs are managed**, and how operators configure both through the Expected Machines table. It is the deep companion to [Ingesting Hosts](ingesting-hosts.md): that page covers the end-to-end ingest flow and the basic `expected_machines.json`; this page covers the per-host and per-NIC knobs (`dpu_mode`, `host_nics`), **what the defaults do when you set nothing**, and how a boot device is chosen and applied behind the scenes.
+This guide explains how NICo decides **which interface a host boots from**, how a host's **DPUs are managed**, and how operators configure both through the Expected Machines table. It is the deep companion to [Ingesting Hosts](ingesting-hosts.md), which covers the end-to-end ingest flow and the basic `expected_machines.json`; this page covers the per-host and per-NIC knobs (`dpu_mode`, `host_nics`), **what the defaults do when you set nothing**, and how a boot device is chosen and applied behind the scenes.
 
 For the DHCP and network-segment substrate these knobs sit on (how a relay's `giaddr` maps to a segment), see [IP and Network Configuration](ip-and-network-configuration.md).
 
@@ -15,9 +15,9 @@ Historically, two separate decisions were conflated into "what kind of host is t
 | Axis | Question | Controlled by |
 |---|---|---|
 | **DPU management** | Does NICo manage this host's DPUs (upgrade them, run agents, serve the host's admin network over the DPU overlay)? | `dpu_mode` |
-| **Boot interface** | Which NIC does the host OS actually boot from and run its management network on? | the host's **primary** interface |
+| **Boot interface** | Which NIC does the host OS boot from and run its management network on? | the host's **primary** interface |
 
-A "normal" host couples them (managed DPU + boot through that DPU). But they are independent: you can keep a host's DPUs **managed** and still boot it from a plain **integrated NIC**. This guide treats the two axes separately, because the configuration knobs are separate.
+A "normal" host couples them (a managed DPU it also boots through). But they are independent: you can keep a host's DPUs **managed** and still boot it from a plain **integrated NIC**. This guide treats the two axes separately, because the configuration knobs are separate.
 
 ### Network segment types
 
@@ -36,6 +36,8 @@ A host's management network lives on one of a few segment types. Which one depen
 
 In NICo the **boot interface and the primary interface are the same thing by construction**: a host's `primary` interface is the one its boot order targets. A boot interface is a `(MAC, Redfish interface id)` pair — the MAC identifies the NIC on the wire; the Redfish id lets NICo set the boot order on the BMC. NICo refers to this pair as the host's `MachineBootInterface`.
 
+NICo keeps both halves because the MAC alone can go stale: a NIC can drop out of the BMC's Redfish inventory (taking its MAC with it) while the interface id remains addressable. When NICo targets a boot operation at the BMC, it tries the MAC first and, when it knows the full pair, retries with the interface id on any error — so boot configuration keeps working when the MAC has vanished from the inventory, as happens after a DPU-to-NIC-mode flip or when a NIC de-enumerates across a reboot.
+
 ---
 
 ## 2. Configuring via Expected Machines and the defaults
@@ -50,7 +52,7 @@ Boot and DPU configuration is **declarative**: you describe the host in the Expe
 - Site-explorer **auto-selects the boot interface**: the lowest-PCI DPU host-PF (the NIC a DPU presents to the host).
 - The host's IP comes from whichever segment its DHCP relay lands in (see [IP and Network Configuration](ip-and-network-configuration.md)).
 
-So a standard DPU host is handled entirely by defaults. The knobs below exist for the hosts that *don't* fit that mold.
+So a standard DPU host is handled entirely by defaults; the knobs below exist for the exceptions.
 
 ### `dpu_mode`
 
@@ -60,7 +62,7 @@ So a standard DPU host is handled entirely by defaults. The knobs below exist fo
 | `nic_mode` / `nic-mode` | DPU hardware is present but treated as a **plain NIC**. Site-explorer explores it but does **not** link or manage it; the host boots on **HostInband**. |
 | `no_dpu` / `no-dpu` | No DPU hardware at all — a plain host NIC on **HostInband**. DPU exploration is skipped entirely. |
 
-**Resolution order:** a per-host `dpu_mode` on the Expected Machine wins; if unset, the site-wide `[site_explorer] dpu_mode` setting applies; if that too is unset, the default is `dpu_mode`.
+**Resolution order:** a per-host `dpu_mode` on the Expected Machine wins; if unset, the site-wide `[site_explorer] dpu_mode` setting applies; if it too is unset, the default is `dpu_mode`.
 
 ### `host_nics` (per-NIC declaration)
 
@@ -74,7 +76,7 @@ The optional `host_nics` array declares specifics for individual host NICs. Each
 | `fixed_ip` / `fixed_mask` / `fixed_gateway` | string | Static IP assignment for the NIC, pre-allocated at upload time. | Dynamic allocation. |
 | `nic_type` | string (legacy) | A free-form segment hint, **superseded by `network_segment_type`**. Kept for backward compatibility only. | — |
 
-> **What `network_segment_type` actually does.** A NIC's segment is normally determined by its DHCP relay: NICo picks the segment whose prefix *contains* the relay address. Where segment prefixes **nest or overlap** — for example a `/27` HostInband segment inside a `/24` underlay — one relay matches several segments. `network_segment_type` narrows that to the segment of the named type. If a relay maps unambiguously to one segment (the common case), this field is unnecessary.
+> **What `network_segment_type` does.** A NIC's segment is normally determined by its DHCP relay: NICo picks the segment whose prefix *contains* the relay address. Where segment prefixes **nest or overlap** — for example a `/27` HostInband segment inside a `/24` underlay — one relay matches several segments. `network_segment_type` narrows the match to the segment of the named type. If a relay maps unambiguously to one segment (the common case), this field is unnecessary.
 
 **JSON** (an Expected Machine entry):
 
@@ -131,7 +133,7 @@ A plain server with one or more host NICs and no DPU. Declare `no_dpu` and mark
 }
 ```
 
-The host boots from that NIC on HostInband and gets its IP from central NICo DHCP.
+The host boots from the declared NIC on HostInband and gets its IP from central NICo DHCP.
 
 ### 3.3 DPU in NIC mode
 
@@ -165,7 +167,7 @@ NICo keeps the DPUs explored, linked, and underlay-addressed (running agents for
 
 ### 3.5 Flipping a DPU to NIC mode
 
-To change a host that's already ingested (e.g. from managed-DPU to NIC mode), update its Expected Machine `dpu_mode`, then force-delete and let it re-ingest so site-explorer re-explores and applies the new mode:
+To change an already-ingested host (e.g. from managed-DPU to NIC mode), update its Expected Machine `dpu_mode`, then force-delete and let it re-ingest so site-explorer re-explores and applies the new mode:
 
 ```bash
 nico-admin-cli -a <api-url> em patch --bmc-mac-address <bmc-mac> --dpu-mode nic-mode
@@ -196,6 +198,7 @@ All of these are **admin-only**; the Forge gRPC service enforces admin authoriza
 
 | admin-cli | Forge RPC | Purpose |
 |---|---|---|
+| `machine boot-interfaces <machine-id>` | `GetMachineBootInterfaces` | Show the boot interface as recorded in every store ([Section 7](#7-the-boot-interface-data-model)), the effective selection, and a divergence flag ([Section 8](#8-verifying-and-troubleshooting)). Read-only. |
 | `managed-host set-primary-interface <host-id> <interface-id> [--reboot]` | `SetPrimaryInterface` | Designate a machine interface as the host's primary/boot interface. **The modern form.** |
 | `managed-host set-primary-dpu <host-id> <dpu-id> [--reboot]` | `SetPrimaryDpu` | Designate a DPU as primary. *Deprecated — prefer `set-primary-interface`.* |
 | `boot-override set <interface-id> [--custom-pxe <f>] [--custom-user-data <f>]` | `SetMachineBootOverride` | Override the iPXE script / cloud-init user-data served at boot. |
@@ -204,6 +207,8 @@ All of these are **admin-only**; the Forge gRPC service enforces admin authoriza
 
 > Setting the DPU-first boot order directly (by MAC) is also exposed as a one-off action through the web UI ([Section 5](#5-web-ui)). Under normal operation the machine-controller sets the boot order automatically during ingestion ([Section 6](#6-behind-the-scenes-how-a-boot-device-is-chosen-and-set)).
 
+> **The direct-to-BMC escape hatch.** The `redfish` command group (`redfish machine-setup`, `redfish set-boot-order-dpu-first`, `redfish is-boot-order-setup`, …) talks to a BMC directly, using operator-supplied credentials and a hand-typed `--boot-interface-mac`. It consults none of the stores in this guide and gets no interface-id fallback. It is useful against a BMC outside NICo's management; for managed hosts, prefer the commands above so the database and the BMC stay in agreement.
+
 ### Ingestion control
 
 | admin-cli | Forge RPC | Purpose |
@@ -228,7 +233,7 @@ The NICo admin web UI (`/admin/…`) is primarily for **visibility**, with a foc
 **Act:**
 
 - **Set DPU First Boot Order** (by MAC) — on a machine or explored endpoint.
-- **Restore Boot Interface** — one-click re-apply of the host's resolved boot interface. It uses the machine's designated primary once managed, or site-explorer's automatic default before that. Handy right after a DPU↔NIC-mode flip.
+- **Restore Boot Interface** — one-click re-apply of the host's resolved boot interface. It uses the machine's designated primary once managed, or site-explorer's automatic default beforehand. Handy right after a DPU↔NIC-mode flip.
 - **Machine Setup** — prepare an endpoint for ingestion (optionally with a boot-interface MAC; Dell endpoints require it).
 - Endpoint controls — **Re-Explore**, **Refresh**, **Clear Last Error**, **Pause/Resume Remediation**, plus power, Secure Boot, lockdown, and BMC-reset actions.
 
@@ -264,26 +269,50 @@ At each boot-config step the controller resolves the target via `load_boot_predi
 2. Otherwise, the **boot prediction** (`pick_boot_prediction`) — used *before* the first lease.
 3. Otherwise, a classification:
    - **AwaitingNic** — a zero-DPU/NIC-mode host whose boot NIC hasn't appeared yet; wait.
-   - **Missing** — a host that *should* have a boot interface (it has DPUs) but doesn't; a fault to investigate.
+   - **Missing** — a DPU host *should* have a boot interface but doesn't; a fault to investigate.
 
-> **Key timing.** A host has no `machine_interfaces` row until its first DHCP lease. Predictions are what let the controller configure boot **before** that lease. Once the host leases and the prediction is promoted to an owned row, the **owned row supersedes** the prediction.
+> **Key timing.** A host has no `machine_interfaces` row until its first DHCP lease. Predictions let the controller configure boot **before** the first lease. Once the host leases and the prediction is promoted to an owned row, the **owned row supersedes** the prediction.
 
 ### Applying the boot order
 
-- `configure_host_bios` (at `WaitingForPlatformConfiguration`) calls Redfish `machine_setup` with the resolved boot interface; on Dell this schedules a BIOS job (`WaitingForBiosJob`).
+- `configure_host_bios` (at `WaitingForPlatformConfiguration`) calls Redfish `machine_setup` with the resolved boot interface. This configures the BIOS, including the **UEFI HTTP-boot device** pointed at the boot NIC; on Dell it schedules a BIOS job (`WaitingForBiosJob`).
 - `PollingBiosSetup` verifies the BIOS settings took.
-- `SetBootOrder` sets the host boot order via Redfish — **DPU-first** for DPU hosts; for zero-DPU/NIC-mode hosts it targets the resolved HostInband interface (a "no DPU" response from the BMC is expected and treated as success).
+- `SetBootOrder` is **check-first**: it verifies the HTTP-boot device and the boot order before writing anything, and a host whose configuration is already in place skips straight to verification. Nothing is re-applied unless something drifted — two BIOS writes never share one pass (on Dell they would collide in a single pending configuration job).
+- If the check finds the HTTP-boot device **reverted** — the boot NIC dropped out of the BMC's Redfish inventory across a reboot, which some NICs do — `SetBootOrder` re-asserts it with `machine_setup` and reboots the host to apply the change first. A `WaitForHttpBootDeviceApplied` substate then polls for up to 10 minutes; each expired window spends one retry from a budget of 3 and returns for a fresh re-assert, and once the budget is exhausted the host is surfaced for manual intervention ([Section 8](#8-verifying-and-troubleshooting)).
+- With the HTTP-boot device in place, the boot order itself is set via Redfish — **DPU-first** for DPU hosts; for zero-DPU/NIC-mode hosts it targets the resolved HostInband interface (a "no DPU" response from the BMC is expected and treated as success).
 - On a reprovision repair, `check_host_boot_config` re-checks BIOS + boot order and only remediates if they drifted.
 
 ---
 
 ## 7. The boot-interface data model
 
-The boot interface flows through three tables: **predicted → managed → retained**.
+A host's boot interface is recorded in **four stores**. Three form the machine's own lineage — **predicted → managed → retained** — and the fourth is the **explored default** site-explorer keeps per BMC endpoint, for hosts no machine owns yet ([Section 7.4](#74-selection-precedence)). `nico-admin-cli machine boot-interfaces` prints all four side by side ([Section 8](#8-verifying-and-troubleshooting)).
+
+```mermaid
+flowchart LR
+    EXPM["expected_machines.host_nics<br/>declared intent — input, not one of the four stores"]
+    subgraph lineage["The machine's own lineage"]
+        PRED[("predicted_machine_interfaces")]
+        OWNED[("machine_interfaces<br/>authoritative once owned")]
+        RET[("retained_boot_interfaces")]
+    end
+    EXPL[("explored_endpoints<br/>automatic default per BMC endpoint")]
+    ACT["boot actions<br/>machine_setup, boot order"]
+
+    EXPM -->|"machine creation mints predictions"| PRED
+    PRED -->|"first DHCP lease promotes"| OWNED
+    OWNED -->|"deletion retains the Redfish id"| RET
+    RET -->|"the next row for the MAC consumes"| OWNED
+    OWNED -->|"owned machines"| ACT
+    PRED -->|"before the first lease"| ACT
+    EXPL -->|"endpoints no machine owns"| ACT
+```
+
+Every exploration pass refreshes the Redfish interface id on owned rows, predictions, and the explored default alike, so all three stay as current as the last healthy exploration.
 
 ### 7.1 Predicted (`predicted_machine_interfaces`)
 
-Site-explorer mints a prediction per declared host NIC **before** the host's first DHCP lease. A prediction carries `machine_id`, `mac_address`, `network_segment_type`, the operator's declared `primary` intent, and the `boot_interface_id` (the Redfish `EthernetInterface.Id`, captured from the exploration report once available). Predictions are what the controller uses to configure boot pre-lease.
+Site-explorer mints a prediction per declared host NIC **before** the host's first DHCP lease. A prediction records `machine_id`, `mac_address`, `network_segment_type`, the operator's declared `primary` intent, and the `boot_interface_id` (the Redfish `EthernetInterface.Id`, captured from the exploration report once available). The controller uses predictions to configure boot pre-lease.
 
 ### 7.2 Managed (`machine_interfaces`) — promotion
 
@@ -301,9 +330,9 @@ The owned table is **Store B**: the authoritative source of truth once a host is
 The Redfish boot-interface id is the one fact a MAC cannot always rediscover after deletion (a DPU/NIC-mode flip can drop the MAC from BMC reports while the id stays stable; a re-ingested host needs to be targeted for boot before a fresh exploration). So:
 
 - On **deletion** of a `machine_interfaces` row, its `boot_interface_id` is **upserted** into `retained_boot_interfaces` (keyed by MAC; newest wins).
-- On **creation** of a new row, any retained id for that MAC is **consumed** and applied — provided it's within the configured `retained_boot_interface_window` (default: no expiry, i.e. retained forever; set a window to bound recycled-MAC reuse).
+- On **creation** of a new row, any retained id for the new row's MAC is **consumed** and applied — provided it's within the configured `retained_boot_interface_window` (default: no expiry, i.e. retained forever; set a window to bound recycled-MAC reuse).
 
-This is what carries a host's boot target across a force-delete / re-ingest gap.
+This preserves a host's boot target across a force-delete / re-ingest gap.
 
 ### 7.4 Selection precedence
 
@@ -321,7 +350,15 @@ The same precedence applies wherever NICo picks a boot interface, over owned row
 
 ## 8. Verifying and troubleshooting
 
-**Check a host's boot interface:**
+**First stop — the four-store view:**
+
+```bash
+nico-admin-cli -a <api-url> machine boot-interfaces <machine-id>
+```
+
+This read-only command prints the machine's boot interface as recorded in all four stores ([Section 7](#7-the-boot-interface-data-model)) side by side — owned rows, predictions, the explored default, and retained pairs (stale ones included) — plus the **effective** boot interface NICo would select, and a `divergent` flag set when the boot-selection signals — the effective owned selection, the explored default, and any primary-flagged prediction — name more than one distinct boot MAC (retained records and non-primary rows don't count toward disagreement). Divergence is normal mid-transition, for example between a mode flip and the re-ingest; persistent divergence on a settled host is worth investigating. For scripting, the global `--output json|yaml` flag applies.
+
+**For a quick primary check:**
 
 ```bash
 nico-admin-cli -a <api-url> managed-host show <machine-id>
@@ -335,6 +372,7 @@ The interfaces section shows each NIC's MAC, segment, and which one is `primary`
 |---|---|
 | `boot_interface_mac_mismatch` (pairing blocker) | The host's boot MAC doesn't match any discovered DPU's pf0 MAC. Expected for an integrated-NIC host — declare the integrated NIC `primary` (see [3.4](#34-boot-an-integrated-nic-while-keeping-the-dpus-managed)); otherwise check the exploration reports. See [Ingesting Hosts → pairing blockers](ingesting-hosts.md#common-blockers-during-host--dpu-pairing). |
 | Host stuck waiting for a boot NIC | A zero-DPU/NIC-mode host whose boot NIC hasn't leased yet (`AwaitingNic`). Confirm the NIC is cabled and DHCP-reachable on its HostInband segment. |
+| `HTTP boot device on host … still not applied after 3 re-asserts` | The boot NIC keeps dropping out of the BMC's Redfish inventory across reboots, so the re-asserted HTTP-boot device never verifies and the retry budget runs out ([Section 6](#6-behind-the-scenes-how-a-boot-device-is-chosen-and-set)). Investigate why the NIC doesn't enumerate (NIC/BMC firmware); once it appears in the BMC's `EthernetInterfaces`, use **Machine Setup** in the web UI to re-assert and reboot. See the [Stuck Objects playbook](../playbooks/stuck_objects/stuck_objects.md) for restarting a failed state machine. |
 | Boot interface wrong after a DPU↔NIC-mode flip | Use **Restore Boot Interface** in the web UI, or re-ingest ([3.5](#35-flipping-a-dpu-to-nic-mode)). |
 | DPU mode "unknown" (`dpu_nic_mode_unknown`) | DPU BMC firmware too old to report mode. Install a fresh DPU OS — see [Ingesting Hosts](ingesting-hosts.md#dpu-related-issues-installing-a-fresh-dpu-os). |