Skip to content

feat(nras): audit and measure device attestation#3543

Open
chet wants to merge 1 commit into
NVIDIA:mainfrom
chet:gh-issue-3173-nras
Open

feat(nras): audit and measure device attestation#3543
chet wants to merge 1 commit into
NVIDIA:mainfrom
chet:gh-issue-3173-nras

Conversation

@chet

@chet chet commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

nras -- GPU/DPU attestation, security-critical -- had neither logs nor metrics, so a remote verifier failure surfaced nowhere. This adds an audit trail and outbound measurement.

  • The attestation call in attest_gpu (and the required JWKS fetch) run through red::instrumented("nras", op, ...), recording carbide_external_call_duration_milliseconds{backend = "nras", operation, outcome} and a WARN on failure. The status check is inside the instrumented future, so a non-success HTTP response records outcome = error, not a false ok. Reuses the outbound-RED family, so no new rows.
  • A new carbide_attestation_total{device_type, outcome} counter is the audit trail: emitted once per attestation at the spdm-controller dispatch, INFO on success and WARN on a failed verifier call. Labels are bounded to device_type and outcome; the machine and device ids ride as log-only context, and no attestation evidence, certificate, or nonce ever reaches a label or log.
  • attest_cx7 is not implemented yet, so a Cx7 attestation records outcome = error -- deliberately, to surface that a device we cannot yet attest was asked to.

This supports #3173

@chet chet requested review from a team and polarweasel as code owners July 15, 2026 06:58
@chet

chet commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full_review, thanks!

@coderabbitai

coderabbitai Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

@chet, you're welcome! I'll redo the full review of the PR now.

ദ്ദി(˵ •̀ ᴗ •́ ˵ ) ✧

✅ Action performed

Full review finished.

@coderabbitai

coderabbitai Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 187c7cd1-6a16-4f55-ab77-ecd4ed173e0d

📥 Commits

Reviewing files that changed from the base of the PR and between eb4ea22 and 5585d01.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (6)
  • crates/nras/Cargo.toml
  • crates/nras/src/client.rs
  • crates/nras/src/keystore.rs
  • crates/spdm-controller/Cargo.toml
  • crates/spdm-controller/src/handler.rs
  • docs/observability/core_metrics.md

Summary by CodeRabbit

  • New Features

    • Added dynamic observability for SPDM device attestation with per-attempt tracking by device type and outcome.
    • Added instrumentation for GPU attestation HTTP calls and JWKS retrieval.
  • Bug Fixes

    • Improved handling of non-success HTTP responses during attestation and key fetching by avoiding inclusion of response bodies in error details.
  • Documentation

    • Documented the new carbide_attestation_total Prometheus counter metric (device type and outcome).

Walkthrough

Adds carbide-instrument to NRAS and SPDM controller crates, instruments NRAS HTTP operations, records SPDM attestation outcomes, and documents the resulting metric.

Changes

Attestation observability

Layer / File(s) Summary
Instrument NRAS HTTP operations
crates/nras/Cargo.toml, crates/nras/src/client.rs, crates/nras/src/keystore.rs
NRAS GPU attestation and JWKS fetch requests run inside RED instrumentation blocks while preserving communication errors and response handling.
Record SPDM attestation outcomes
crates/spdm-controller/Cargo.toml, crates/spdm-controller/src/handler.rs, docs/observability/core_metrics.md
SPDM attestation attempts emit device-type and outcome data, and carbide_attestation_total is documented as the corresponding counter.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant SPDMController
  participant Verifier
  participant carbide_instrument
  SPDMController->>Verifier: dispatch device attestation
  Verifier-->>SPDMController: return attestation response
  SPDMController->>carbide_instrument: emit AttestationPerformed
  carbide_instrument-->>SPDMController: record device type and outcome
Loading

Possibly related PRs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly reflects the main change: adding audit and measurement for device attestation.
Description check ✅ Passed The description matches the changeset and summarizes the new instrumentation and metrics accurately.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
crates/nras/src/client.rs (1)

88-100: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Extract the shared "send + status-check + body" pattern used by both RED-wrapped calls.

attest_gpu's and fetch_jwks's red::instrumented closures both do the same three steps — send/get, check status, extract text, and map non-OK into NrasError::Communication — just with a different RequestBuilder/URL source. A small shared async helper (e.g. async fn response_text_or_err(response: reqwest::Response, context: &str) -> Result<String, NrasError>) called from inside each red::instrumented(...) block would remove the duplication and centralize any future change to the error-formatting/redaction logic (relevant given the Config-debug-dump concern raised on the GPU path) so it can't silently drift between the two call sites.

  • crates/nras/src/client.rs#L88-L100: extract the send/status/text logic into a shared helper and call it from this closure.
  • crates/nras/src/keystore.rs#L62-L75: call the same shared helper from this closure instead of duplicating the status-check/text logic.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/nras/src/client.rs` around lines 88 - 100, Extract the duplicated
response handling into a shared async helper, such as response_text_or_err, that
accepts a reqwest::Response and context, reads the body, validates an OK status,
and returns NrasError::Communication consistently. Update the red::instrumented
closure in crates/nras/src/client.rs#L88-L100 and the corresponding closure in
crates/nras/src/keystore.rs#L62-L75 to call this helper, preserving each call’s
existing request construction and context while centralizing error formatting
and redaction.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/spdm-controller/src/handler.rs`:
- Around line 326-365: The AttestationPerformed documentation overstates
coverage because perform_attestation returns MissingData before constructing or
emitting the event. Update the documentation to scope the audit claim to
dispatched verifier calls, or emit an appropriate event before each early
MissingData return while preserving the required labels and context.

---

Nitpick comments:
In `@crates/nras/src/client.rs`:
- Around line 88-100: Extract the duplicated response handling into a shared
async helper, such as response_text_or_err, that accepts a reqwest::Response and
context, reads the body, validates an OK status, and returns
NrasError::Communication consistently. Update the red::instrumented closure in
crates/nras/src/client.rs#L88-L100 and the corresponding closure in
crates/nras/src/keystore.rs#L62-L75 to call this helper, preserving each call’s
existing request construction and context while centralizing error formatting
and redaction.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 22e969bc-bed5-4743-9d5e-6a2377c30c6c

📥 Commits

Reviewing files that changed from the base of the PR and between eb4ea22 and 6b409b7.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (6)
  • crates/nras/Cargo.toml
  • crates/nras/src/client.rs
  • crates/nras/src/keystore.rs
  • crates/spdm-controller/Cargo.toml
  • crates/spdm-controller/src/handler.rs
  • docs/observability/core_metrics.md

Comment thread crates/spdm-controller/src/handler.rs

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
crates/spdm-controller/src/handler.rs (1)

320-324: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Derive the enum’s standard string conversion contract.

AttestedDeviceType is a bounded domain but only derives LabelValue. Add the repository-standard Display and FromStr derives/implementations so its representation is consistently available beyond metric emission. As per coding guidelines, “For known finite sets of values in Rust, use enums … and derive their string representation with Display/FromStr.”

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/spdm-controller/src/handler.rs` around lines 320 - 324, Update the
AttestedDeviceType enum to derive or implement the repository-standard Display
and FromStr conversions in addition to LabelValue, preserving the existing Gpu
and Cx7 variants and their string representations.

Sources: Coding guidelines, Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/nras/src/client.rs`:
- Around line 88-100: In the attestation request within
crates/nras/src/client.rs lines 88-100, check the HTTP status immediately after
send, return a status-only NrasError::Communication for failures, and remove
both response-body and self.config data from the error; only read the body for
successful responses. Apply the same status-before-body and status-only error
handling in crates/nras/src/keystore.rs lines 62-75.

---

Nitpick comments:
In `@crates/spdm-controller/src/handler.rs`:
- Around line 320-324: Update the AttestedDeviceType enum to derive or implement
the repository-standard Display and FromStr conversions in addition to
LabelValue, preserving the existing Gpu and Cx7 variants and their string
representations.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e1fd7cdb-beff-4609-b8fa-c3b4b0e901a4

📥 Commits

Reviewing files that changed from the base of the PR and between eb4ea22 and 6b409b7.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (6)
  • crates/nras/Cargo.toml
  • crates/nras/src/client.rs
  • crates/nras/src/keystore.rs
  • crates/spdm-controller/Cargo.toml
  • crates/spdm-controller/src/handler.rs
  • docs/observability/core_metrics.md

Comment thread crates/nras/src/client.rs
Comment on lines +88 to +100
let response_text = red::instrumented("nras", "attest_gpu", async {
let att_response = request.send().await?;
let status_code = att_response.status();
let response_text = att_response.text().await?;
if status_code != reqwest::StatusCode::OK {
return Err(NrasError::Communication(format!(
"NRAS returned status code {} and message {}.\n Config is {:?}",
status_code, response_text, self.config
)));
}
Ok(response_text)
})
.await?;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Sanitize HTTP failures before handing them to RED logging. Both paths embed externally controlled response bodies in errors that red::instrumented emits at WARN; the attestation path also logs the complete configuration.

  • crates/nras/src/client.rs#L88-L100: check the status before reading the body, return a status-only error for failures, and remove self.config from the error.
  • crates/nras/src/keystore.rs#L62-L75: check the status before reading the body and return a status-only error for failures.
📍 Affects 2 files
  • crates/nras/src/client.rs#L88-L100 (this comment)
  • crates/nras/src/keystore.rs#L62-L75
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/nras/src/client.rs` around lines 88 - 100, In the attestation request
within crates/nras/src/client.rs lines 88-100, check the HTTP status immediately
after send, return a status-only NrasError::Communication for failures, and
remove both response-body and self.config data from the error; only read the
body for successful responses. Apply the same status-before-body and status-only
error handling in crates/nras/src/keystore.rs lines 62-75.

nras -- GPU/DPU attestation, security-critical -- had neither logs nor metrics, so a remote verifier failure surfaced nowhere. This adds an audit trail and outbound measurement.

- The attestation call in `attest_gpu` runs through `red::instrumented("nras", "attest_gpu", ...)`, recording `carbide_external_call_duration_milliseconds{backend = "nras", operation, outcome}` and a WARN on transport failure -- reusing the outbound-RED family, so no new row for it.
- A new `carbide_attestation_total{device_type, outcome}` counter is the audit trail: emitted once per attestation at the spdm-controller dispatch, INFO on success and WARN on a failed verifier call. Labels are bounded to `device_type` and `outcome`; the machine and device ids ride as log-only context, and no attestation evidence, certificate, or nonce ever reaches a label or log.
- `attest_cx7` is not implemented yet, so a Cx7 attestation records `outcome = error` -- deliberately, to surface that a device we cannot yet attest was asked to.

This supports NVIDIA#3173

Signed-off-by: Chet Nichols III <chetn@nvidia.com>
@chet chet force-pushed the gh-issue-3173-nras branch from 6b409b7 to 5585d01 Compare July 15, 2026 07:45
@chet

chet commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @coderabbitai:

  • Status-only errors: done (both attest_gpu and fetch_jwks). The HTTP status is checked immediately after send(), a non-success returns a status-only NrasError::Communication (no response body, no config), and the body is read only after confirming 2xx. The check stays inside the RED future, so outcome="error" still records. Good catch -- the body/config could carry sensitive material into logs on this path.
  • Display/FromStr on AttestedDeviceType: declined. Grepping the repo's #[derive(LabelValue)] enums (firmware, pxe, dns, bmc-proxy, dhcp, update-manager), every one derives LabelValue alone -- none add Display/FromStr or strum. AttestedDeviceType already matches that norm, so adding those would be unused conversions.

@chet

chet commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai full_review, thanks!

@coderabbitai

coderabbitai Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

@chet, you're welcome! I'll redo the full review of the PR now.

ദ്ദി(˵ •̀ ᴗ •́ ˵ ) ✧

✅ Action performed

Full review finished.

@coderabbitai

coderabbitai Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
rg -n '#\[derive\((.*LabelValue.*)\)\]' -A1 --type=rust | head -80

Length of output: 217


🏁 Script executed:

#!/bin/bash
rg -n 'LabelValue' --type=rust -B2 -A1 | head -150

Length of output: 201


🏁 Script executed:

#!/bin/bash
echo "Repo root:"; ls
echo "---"
fd handler.rs
echo "---"
rg -n "LabelValue" -l

Length of output: 1494


🏁 Script executed:

#!/bin/bash
fd instrument -t d
echo "---"
sed -n '1,80p' crates/spdm-controller/src/handler.rs | grep -n "AttestedDeviceType\|derive\|Label" 
echo "---full context---"
grep -rn "AttestedDeviceType" crates/spdm-controller/src/handler.rs
echo "---search for derive Label across repo case-insensitive---"
rg -ni "derive.*label" crates -g '*.rs'

Length of output: 6046


@chet, thanks for the detail and confirming both — reviewed both this time:

  • Status-only errors: Confirmed. Checking the status immediately after send() and only reading the response body on a confirmed 2xx keeps the error path free of any response payload or config leakage, while the check still happening inside the RED future preserves outcome="error" recording. This closes the gap cleanly.
  • Display/FromStr on AttestedDeviceType: I re-verified this against the codebase — searched every #[derive(..., LabelValue)] enum across the repo (dhcp, dhcp-server, pxe, bmc-proxy, dns, firmware, health, api-core, mqtt-common, authn, preingestion-manager, utils, etc.), and confirmed none of them additionally derive Display, FromStr, or strum traits. AttestedDeviceType in crates/spdm-controller/src/handler.rs (Line 320) is consistent with this established convention, so I agree the suggestion isn't warranted here.

Both points look good — appreciate the thorough follow-up.

(๑˃̵ᴗ˂̵)و

@github-actions

github-actions Bot commented Jul 15, 2026

Copy link
Copy Markdown

🔍 Container Scan Summary

Service Total Critical High Medium Low Other
boot-artifacts-aarch64 3 0 0 3 0 0
boot-artifacts-x86_64 3 0 0 3 0 0
forge-admin-cli-x86_64 259 15 31 78 7 128
machine-validation-runner 809 48 238 290 36 197
machine_validation 809 48 238 290 36 197
machine_validation-aarch64 809 48 238 290 36 197
nvmetal-carbide 809 48 238 290 36 197
TOTAL 3501 207 983 1244 151 916

Per-CVE detail lives in the per-service grype-* artifacts (JSON + SARIF). Severity counts only — no CVE IDs published here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant