Automate lifecycle (rotation, revocation) of managed (owned) passwords

[sabbanis]

NICo (BMM) manages several passwords (i.e. passwords that NICo is authoritative for):

• BMC (Host & DPU) passwords
• UEFI passwords
• IB/NIC Lockdown keys

NICo must support rotating passwords (i.e. changing them on the device) and when possible changing them while a machine is in use, without requiring a reboot or the user releasing the machine.

[chet]

when possible changing them while a machine is in use, without requiring a reboot or the user releasing the machine.

We need to make sure we design with the assumption that certain passwords won't be able to be dropped until the user releases a machine.

An example is SuperNIC lockdown keys. We use a KDF that takes the shared secret + some context to produce the key, but once a card is locked, we need to keep that shared secret around so we can unlock the card once the tenant releases the machine. We can produce a NEW shared secret, and all NEW lockdown keys can be derived from it, but not until the last card is unlocked on a previous generation of a shared secret can we actually sweep the old secret.

And yes, for ASTRA we actually don't have to worry about this, because the CX9s are locked out by design, but as of now that's only specific to ASTRA + E/W Spectrum-X.

[mxh-0xbb]

Some relevant commands:

carbide-admin-cli set site-wide-password
or each bmc; do carbide-admin-cli credential delete-bmc && carbide-admin-cli-redfish change-bmc-password && carbide-admin-cli credential create-bmc; done

[ddejong-spec]

crates/secrets/src/credentials.rs enumerates the types of credentials we store as CredentialKey, and is used to determine the Vault key so we can be reasonable confident that there is nothing missing:

pub enum CredentialKey {
    DpuSsh { machine_id: MachineId },
    DpuHbn { machine_id: MachineId },
    DpuRedfish { credential_type: CredentialType },
    HostRedfish { credential_type: CredentialType },
    UfmAuth { fabric: String },
    DpuUefi { credential_type: CredentialType },
    HostUefi { credential_type: CredentialType },
    BmcCredentials { credential_type: BmcCredentialType },
    ExtensionService { service_id: String, version: String },
    NmxM { nmxm_id: String },
    RackFirmware { firmware_id: String },
    SwitchNvosAdmin { bmc_mac_address: MacAddress },
}

(Also soon to be merged MqttAuth, looks like it's manually managed at the moment)

pub enum CredentialType {
    DpuHardwareDefault,
    HostHardwareDefault { vendor: bmc_vendor::BMCVendor },
    SiteDefault,
}

pub enum BmcCredentialType {
    // Site Wide Root Credentials
    SiteWideRoot,
    // BMC Specific Root Credentials
    BmcRoot { bmc_mac_address: MacAddress },
    // BMC Specific Forge-Admin Credentials
    BmcForgeAdmin { bmc_mac_address: MacAddress },
}

We have to be cautious about reuse; for example, BmcCredentials{credential_type: BmcCredentialtType::SiteWideRoot} is used by multiple things (including the stuff Chet cautioned about), and if we're trying to update that then we must be careful to reach all of those.

The process being interrupted in the middle could require a large amount of manual work to fix (or even be impossible in some implementations), so we need it to be done in a safe manner. Current thoughts on how to do this: When rotating keys, we should first set a flag in the database that we are have started to rotate; this includes a timestamp of expected completion and the details of the request, not including the secrets themselves. After this, we save the new credentials in vault using a key that is a predictable unique modification of the current key, then change the password, then update the original vault key, then remove the rotating flag, and finally remove the rotating vault key. When we start, we must look for the rotating flag; if it was set, first we wait until after the completion time, then if it is still set we redo the flag and we have to go to try and log in with the old and new keys; if the old key works, we have to update it.

The rotation flag table should always be accessed with a lock. If it is already set, we cannot proceed until it goes away. We need to decide if parallel requests should return an error for the new requests or if they wait for completion. If they wait, we should probably just retry once a second; with multiple carbide-api instances they can't directly communicate. We could have something with subscriptions, but that's probably overengineered.

All of this should give us a method of doing updates that are reliable in the face of potential restarts and potential multiple instances of carbide-api.

Additionally, we need to be cautious that the secrets are not captured by any automatic logging.

[mxh-0xbb]

CredentialKey seems like a good way to start the catalog all the (kinds of) keys that might need rotation. It seems like there might be a lot of individual credentials: DpuSsh { machine_id: MachineId } certainly seems to suggest a different key for each machine under management.

I think there might be some more difficult scenarios to cover:

• Some rotations might not be able to complete until all tenants have released their instances, or at least permitted a restart, and all machines have come back from break-fix. These rotations could take a very long time (weeks!) to complete, and in the meantime other code would need to be able to use the appropriate credentials.
• Following on from the above, rotations would need to gracefully handle interruptions and restarts.
• It could even happen that a credential needed to be rotated before it had fully rolled out. This would be extraordinary, but it's better to think about it now than later.

I believe an SRD is indicated here.