Add `sudo_unstable_metrics` JSON-RPC function by lrubasze · Pull Request #3318 · paritytech/smoldot

lrubasze · 2026-07-31T10:07:05Z

Adds sudo_unstable_metrics, returning a snapshot of internal counters and gauges as a flat list of families (name, type, entries with labels). The set of metrics and labels is unstable and may change between versions.

Covered: network connections and discovery drops (process-wide); per-chain request counts/durations per protocol, peer bans by reason, gossip peers, block/finality verification outcomes, best/finalized/warp heights, runtime compilations (including cold-start ones), dropped transactions, JSON-RPC request count.

Notes:

Counters are relaxed atomics, incremented next to the existing log calls. Label values derive from enums via strum; tests pin the exact label sets.
Breaking: ban_and_disconnect now takes a BanReason enum instead of &'static str (network_service is public API).

Use in e2e-tests

The chainhead e2e test can sample the new function during a run and render the time series as an HTML report:

SMOLDOT_METRICS_OUT=metrics.json ./run_chainhead_test.sh kusama-ah

This polls sudo_unstable_metrics on every chain every 5s (SMOLDOT_METRICS_INTERVAL_MS to change), writes the dump to metrics.json, and renders metrics.html (self-contained, no dependencies): block heights, warp progress, gossip peers, per-protocol request rate/failures/duration, runtime compilations, peer bans and discovery drops by reason, connection churn. The dump is written even when the test fails, so a bad run still produces a report - one paseo run with a warp stall was diagnosed straight from the charts (gossip peers at 0, handshake rate dropping to zero). Entirely opt-in: without SMOLDOT_METRICS_OUT nothing is sampled.

Add a metrics module with atomic counters/gauges: one NetworkMetrics per process and one ChainMetrics per chain, threaded through the network/sync/runtime/transactions/json-rpc services and incremented next to the existing log statements. Not exposed yet.

Returns a snapshot of the internal metric families as JSON. The list of metrics and their labels is unstable. Light-base only; full-node answers with the existing not-implemented error.

Add networkRequestSecondsTotal{protocol} count+sum next to the existing request outcome counters. Duration is client-observed (includes background-task queueing), measured in the NetworkServiceChain foreground wrappers; the PhantomData marker becomes a real platform field to get access to the clock.

syncWarpSyncHeight follows the last verified warp fragment; syncWarpSyncTargetHeight is the peer-claimed finalized height captured at the WarpAhead commit. Both read 0 until first set.

Set SMOLDOT_METRICS_OUT=<file.json> on run_chainhead_test.sh to poll sudo_unstable_metrics during the run (5s default) and render a self-contained HTML report (heights incl. warp progress, peers, request rates/durations, connection churn). The sampler polls the muxed chain through JsonRpcMux and rpc-managed chains through a new id-routed sendRpcOutOfBand, so it cannot steal the test's messages; the dump is written in a finally block so failing runs keep their graphs.

…mple

socket-security · 2026-07-31T10:07:26Z

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	cargo/strum@0.27.2

View full report

lrubasze · 2026-07-31T10:10:34Z

Metrics example generated with below command:

SMOLDOT_METRICS_OUT=metrics.json ./run_chainhead_test.sh kusama-ah

Returned json with snapshot

  {
    "t": 1785492166420,
    "chain": "para",
    "metrics": [
      {
        "name": "networkConnectionsStartedTotal",
        "type": "counter",
        "entries": [
          {
            "value": 123
          }
        ]
      },
      {
        "name": "networkConnectionsHandshakesFinishedTotal",
        "type": "counter",
        "entries": [
          {
            "value": 22
          }
        ]
      },
      {
        "name": "networkConnectionsShutdownsTotal",
        "type": "counter",
        "entries": [
          {
            "value": 112
          }
        ]
      },
      {
        "name": "networkDiscoveryAddressesDroppedTotal",
        "type": "counter",
        "entries": [
          {
            "labels": {
              "reason": "peer-id-mismatch"
            },
            "value": 0
          },
          {
            "labels": {
              "reason": "not-supported"
            },
            "value": 8
          },
          {
            "labels": {
              "reason": "invalid"
            },
            "value": 0
          }
        ]
      },
      {
        "name": "networkRequestsTotal",
        "type": "counter",
        "entries": [
          {
            "labels": {
              "outcome": "success",
              "protocol": "blocks"
            },
            "value": 5
          },
          {
            "labels": {
              "outcome": "failure",
              "protocol": "blocks"
            },
            "value": 0
          },
          {
            "labels": {
              "outcome": "success",
              "protocol": "warp-sync"
            },
            "value": 0
          },
          {
            "labels": {
              "outcome": "failure",
              "protocol": "warp-sync"
            },
            "value": 0
          },
          {
            "labels": {
              "outcome": "success",
              "protocol": "storage-proof"
            },
            "value": 1
          },
          {
            "labels": {
              "outcome": "failure",
              "protocol": "storage-proof"
            },
            "value": 0
          },
          {
            "labels": {
              "outcome": "success",
              "protocol": "call-proof"
            },
            "value": 2
          },
          {
            "labels": {
              "outcome": "failure",
              "protocol": "call-proof"
            },
            "value": 0
          }
        ]
      },
      {
        "name": "networkRequestSecondsTotal",
        "type": "counter",
        "entries": [
          {
            "labels": {
              "protocol": "blocks"
            },
            "value": 0.588458
          },
          {
            "labels": {
              "protocol": "warp-sync"
            },
            "value": 0
          },
          {
            "labels": {
              "protocol": "storage-proof"
            },
            "value": 1.112125
          },
          {
            "labels": {
              "protocol": "call-proof"
            },
            "value": 0.039935
          }
        ]
      },
      {
        "name": "networkPeerBansTotal",
        "type": "counter",
        "entries": [
          {
            "labels": {
              "reason": "bad-block"
            },
            "value": 0
          },
          {
            "labels": {
              "reason": "bad-block-announce"
            },
            "value": 0
          },
          {
            "labels": {
              "reason": "bad-child-trie-root"
            },
            "value": 0
          },
          {
            "labels": {
              "reason": "bad-grandpa-commit"
            },
            "value": 0
          },
          {
            "labels": {
              "reason": "bad-justification"
            },
            "value": 0
          },
          {
            "labels": {
              "reason": "bad-merkle-proof"
            },
            "value": 0
          },
          {
            "labels": {
              "reason": "bad-warp-sync-fragment"
            },
            "value": 0
          },
          {
            "labels": {
              "reason": "invalid-call-proof"
            },
            "value": 0
          },
          {
            "labels": {
              "reason": "blocks-request-failed"
            },
            "value": 0
          },
          {
            "labels": {
              "reason": "call-proof-request-failed"
            },
            "value": 0
          },
          {
            "labels": {
              "reason": "child-storage-request-failed"
            },
            "value": 0
          },
          {
            "labels": {
              "reason": "storage-request-failed"
            },
            "value": 0
          },
          {
            "labels": {
              "reason": "warp-sync-request-failed"
            },
            "value": 0
          }
        ]
      },
      {
        "name": "networkGossipPeersConnected",
        "type": "gauge",
        "entries": [
          {
            "value": 4
          }
        ]
      },
      {
        "name": "syncBlocksVerifiedTotal",
        "type": "counter",
        "entries": [
          {
            "labels": {
              "outcome": "success"
            },
            "value": 35
          },
          {
            "labels": {
              "outcome": "failure"
            },
            "value": 0
          }
        ]
      },
      {
        "name": "syncFinalityProofsVerifiedTotal",
        "type": "counter",
        "entries": [
          {
            "labels": {
              "outcome": "success"
            },
            "value": 0
          },
          {
            "labels": {
              "outcome": "failure"
            },
            "value": 0
          }
        ]
      },
      {
        "name": "syncWarpFragmentsVerifiedTotal",
        "type": "counter",
        "entries": [
          {
            "value": 0
          }
        ]
      },
      {
        "name": "syncBestBlockHeight",
        "type": "gauge",
        "entries": [
          {
            "value": 19687690
          }
        ]
      },
      {
        "name": "syncFinalizedBlockHeight",
        "type": "gauge",
        "entries": [
          {
            "value": 19687672
          }
        ]
      },
      {
        "name": "syncWarpSyncHeight",
        "type": "gauge",
        "entries": [
          {
            "value": 0
          }
        ]
      },
      {
        "name": "syncWarpSyncTargetHeight",
        "type": "gauge",
        "entries": [
          {
            "value": 0
          }
        ]
      },
      {
        "name": "runtimeCompilationsTotal",
        "type": "counter",
        "entries": [
          {
            "value": 1
          }
        ]
      },
      {
        "name": "runtimeCompilationErrorsTotal",
        "type": "counter",
        "entries": [
          {
            "value": 0
          }
        ]
      },
      {
        "name": "runtimeCompilationSecondsTotal",
        "type": "counter",
        "entries": [
          {
            "value": 0.148646
          }
        ]
      },
      {
        "name": "runtimeCacheHitsTotal",
        "type": "counter",
        "entries": [
          {
            "value": 0
          }
        ]
      },
      {
        "name": "transactionsDroppedTotal",
        "type": "counter",
        "entries": [
          {
            "labels": {
              "reason": "gap-in-chain"
            },
            "value": 0
          },
          {
            "labels": {
              "reason": "max-pending-transactions-reached"
            },
            "value": 0
          },
          {
            "labels": {
              "reason": "invalid"
            },
            "value": 0
          },
          {
            "labels": {
              "reason": "validate-error"
            },
            "value": 0
          }
        ]
      },
      {
        "name": "jsonrpcRequestsTotal",
        "type": "counter",
        "entries": [
          {
            "value": 75
          }
        ]
      }
    ]
  }

Rendered metrics using 5s snapshots as above

lexnv · 2026-07-31T10:56:44Z

+/// Metrics of the network service. One instance per process, shared between all chains.
+#[derive(Debug, Default)]
+pub struct NetworkMetrics {
+    pub connections_started: Counter,


These are peers that connected on the webrtc and the connections_handshakes_finished represent the peers that completed the handshakes? 🤔

These counters apply to any transport type (websocket, WebRTC, TCP).
connection_started is incremented upon each dial attempt, no matter the result.
connections_handshakes_finished is the subset of those that completed the handshake.
And the counters are per-connection, not per-peer - one peer can account for several attempts.

I'll update the doc comment.
Thanks!

lexnv · 2026-07-31T11:28:11Z

+#[derive(Debug, Clone, strum::EnumDiscriminants)]
+#[strum_discriminants(name(DropReasonKind))]
+#[strum_discriminants(derive(strum::EnumIter, strum::IntoStaticStr))]
+#[strum_discriminants(strum(serialize_all = "kebab-case"))]


This is for label info in grafana?

Sort of - it's the reason label of the transactionsDroppedTotal metric in the sudo_unstable_metrics response (not Grafana-specific; anything consuming the metrics sees it, e.g. a Prometheus adapter or the e2e HTML report). EnumDiscriminants generates a payload-free DropReasonKind usable as a label, EnumIter gives the snapshot the full label set (zero counts included), IntoStaticStr + kebab-case turn variant names into the label strings, and the disabled variants (Finalized = success path, Crashed) are deliberately not counted. A test pins the exact label strings so a rename can't silently change the metric.

Added docs for this. Thanks!

lexnv

Looks good on the Rust side (dont have much context on JS realm) 🙏

michalkucharczyk · 2026-08-07T09:37:20Z

    }

+    /// Returns the process-wide network metrics.
+    pub fn metrics(&self) -> &Arc<metrics::NetworkMetrics> {


do we actually need Arc here?

michalkucharczyk · 2026-08-07T10:49:02Z

+        "networkRequestSecondsTotal", { protocol: p },
+        "networkRequestsTotal", { protocol: p, outcome: "success" },
+      ),
+    })),


networkRequestSecondsTotal includes failures, but we only divide by success.

Related code:

smoldot/light-base/src/metrics.rs

Lines 97 to 105 in 6fabafe

pub fn observe(&self, is_success: bool, duration: core::time::Duration) {

if is_success {

self.success.inc();

} else {

self.failure.inc();

}

self.duration_ms

.add(u64::try_from(duration.as_millis()).unwrap_or(u64::MAX));

}

michalkucharczyk · 2026-08-07T10:57:19Z

-        rx.await.unwrap()
+        let result = rx.await.unwrap();
+        self.metrics
+            .storage_proof_requests


child_storage_proof_request ? Or do we intentionally merge them?

michalkucharczyk · 2026-08-07T10:59:00Z

+)]
+#[strum(serialize_all = "kebab-case")]
+pub enum BanReason {
+    BadBlock,


never used?

michalkucharczyk · 2026-08-07T11:10:26Z

+
+// 32-bit atomics so that the metrics also compile on targets without 64-bit
+// atomics (e.g. `thumbv7m-none-eabi`); durations are tracked in milliseconds
+// to make the range acceptable.


Nit: not sure if limiting to 32-bits on all platforms is OK here. Not saying it shall be 64bit - but maybe this deserves a second though if we can do something smarter here. E.g. 2^32 ~ 51 days, technically duration could accumulate to such number. But maybe it is ok.

michalkucharczyk

Skimmed the implementation, looks good. Left some nits.

lrubasze added 18 commits July 31, 2026 08:55

json-rpc: add sudo_unstable_metrics function (#3285)

fa71113

Returns a snapshot of the internal metric families as JSON. The list of metrics and their labels is unstable. Light-base only; full-node answers with the existing not-implemented error.

wasm-node: add sudo_unstable_metrics test and changelog entry

baa6e29

light-base: add warp sync progress gauges (#3285)

40a0391

syncWarpSyncHeight follows the last verified warp fragment; syncWarpSyncTargetHeight is the peer-claimed finalized height captured at the WarpAhead commit. Both read 0 until first set.

light-base: align metric labels with #3285 samples

f82c721

e2e-tests: read labeled verify metrics in HTML report

34ef60a

e2e-tests: add warp progress and runtime compilation charts

04d5b05

light-base: add reason label to networkPeerBansTotal

f2c4c74

light-base: kebab-case metric label values

8a83af7

e2e-tests: stacked ban and discovery-drop charts in metrics report

00b64b8

e2e-tests: network section rename + always-visible per-chain ban charts

756992d

e2e-tests: always show legend on stacked charts

5f08a75

light-base: count cold-start runtime compilations

28755be

cleanup: tighten comments added on this branch

a4e44a9

e2e-tests: full block numbers in metrics report

a55a306

e2e-tests: show warp completion step when warp finishes within one sa…

e040b40

…mple

light-base: 32-bit metric atomics for no-std targets, durations in ms.

415bcd4

lrubasze requested a review from a team July 31, 2026 10:37

lexnv reviewed Jul 31, 2026

View reviewed changes

lexnv approved these changes Jul 31, 2026

View reviewed changes

lrubasze added 4 commits July 31, 2026 11:54

light-base: document NetworkMetrics connection counters

d3d6815

light-base: rename connections started metric to dialed

767dc68

light-base: comment purpose of strum derives on DropReason

2e6f5d9

light-base: consolidate strum label documentation on MetricLabel

6fabafe

michalkucharczyk reviewed Aug 7, 2026

View reviewed changes

michalkucharczyk approved these changes Aug 7, 2026

View reviewed changes

	pub fn observe(&self, is_success: bool, duration: core::time::Duration) {
	if is_success {
	self.success.inc();
	} else {
	self.failure.inc();
	}
	self.duration_ms
	.add(u64::try_from(duration.as_millis()).unwrap_or(u64::MAX));
	}

Uh oh!

Conversation

lrubasze commented Jul 31, 2026

Use in e2e-tests

Uh oh!

socket-security Bot commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lrubasze commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lrubasze Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lexnv left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

michalkucharczyk Aug 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

michalkucharczyk left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

socket-security Bot commented Jul 31, 2026 •

edited

Loading

lrubasze commented Jul 31, 2026 •

edited

Loading

lrubasze Jul 31, 2026 •

edited

Loading

lexnv left a comment •

edited

Loading

michalkucharczyk Aug 7, 2026 •

edited

Loading