A DGX Spark TUI that takes the heat.
A terminal dashboard for small NVIDIA DGX Spark clusters - host, GPU, and vLLM inference metrics for every node, live. No Prometheus, no Grafana, no browser.
Built with dado (a tcell-based TUI
toolkit). Inspired by paul-aviles/NVIDIA-DGX-Spark-Dashboard.
- Top bar aggregates the fleet: nodes up, GPU count, total power draw, peak GPU temperature
- Per-node health score (0-100) folding in CPU, memory, disk, thermal, and throttle pressure
- Nodes that stop responding flip to
DOWNwith the error and recover automatically - outages render as dips in the timelines, not blank charts
- 2×2 KPI cards - util, temp, power, unified memory % - each with a trend arrow and a full-width sparkline, thresholds tuned to the GB10 envelope (temp warns at 80 °C, power at 110 W, memory at 80 %)
- A red
THROTTLINGbadge when a busy GPU's SM clock sags below its observed peak - the tell for thermal/power throttling - Memory is shown once, unified: on GB10, "VRAM" and system RAM are the same physical LPDDR5X pool
- Served model, running/waiting request counts (waiting turns yellow: backpressure), KV-cache %, and live generation tok/s as a solid block-area chart
- A latency detail line derived from vLLM's histograms, averaged over each poll window:
- TTFT - time-to-first-token, what a user feels before streaming starts
- ITL - inter-token latency, the streaming smoothness
- prefill tok/s - prompt-ingest load
- preemptions - anything above zero means the KV cache is thrashing
- Core × time heatmap - every core's utilization history in one dense map (idle fades to the background, load walks green → yellow → red)
- Host pressure line: CPU %, load vs. core count, root filesystem %
- Network rx/tx and disk read/write as stacked block-bar bands with live rates
- Scrapes
node_exporter, NVIDIAdcgm-exporter, and vLLM's/metricsover plain HTTP - Built-in
deploy/teardown/healthsubcommands manage the exporter containers over SSH - 26 built-in themes, switchable at runtime
brew install galaxy-io/tap/sparktopgo install github.com/galaxy-io/sparktop/cmd/sparktop@latestEvery release ships linux/darwin/windows × amd64/arm64 archives and
a Debian package on the releases page -
the .deb installs cleanly on the DGX Spark nodes themselves.
Needs Go 1.25+ and Task. Builds anywhere - your laptop, or one of the Spark nodes.
git clone https://github.com/galaxy-io/sparktop.git
cd sparktop
task build # → ./sparktop (or: go build -o sparktop ./cmd/sparktop)
task build-arm # → ./sparktop-linux-arm64, for the DGX Spark nodesThese are the only things that run on the nodes (two small containers).
Prereqs per node: Docker + Compose plugin, and the NVIDIA Container Toolkit
(DGX OS ships with it; otherwise see the
install guide;
verify with docker run --rm --gpus all ubuntu nvidia-smi).
From your workstation:
sparktop deploy me@spark-01 me@spark-02This SSHes to each target, uploads the embedded docker-compose.yml
(internal/exporters/docker-compose.yml in this repo, if you'd rather deploy
by hand), pulls the images, and brings the stack up. Then check reachability:
sparktop health spark-01 spark-02Tear it down later with sparktop teardown me@spark-01 me@spark-02
(add --purge to also remove ~/sparktop-exporters on each node).
No config file needed to try it:
sparktop -nodes spark-01=192.168.1.101,spark-02=192.168.1.102
# serving LLMs? add the vLLM metrics port:
sparktop -nodes spark-01=192.168.1.101,spark-02=192.168.1.102 -vllm-port 8000For everyday use, create a config once and then plain sparktop works:
mkdir -p ~/.config/sparktop
curl -fsSL https://raw.githubusercontent.com/galaxy-io/sparktop/main/config.yaml.example \
-o ~/.config/sparktop/config.yaml
$EDITOR ~/.config/sparktop/config.yaml # put your node IPs in it
sparktop| Key | Action |
|---|---|
r |
Refresh now |
t |
Cycle theme (26 built-in dado themes) |
q / Ctrl-C |
Quit |
Each flag overrides the config file:
| Flag | Description |
|---|---|
-config <file> |
Explicit config file path |
-nodes name=host,… |
Comma-separated node list |
-interval 2s |
Poll cadence |
-theme <name> |
Theme name |
-vllm-port 8000 |
Scrape vLLM /metrics on this port for every node (use per-node vllm_port: in the config for mixed setups) |
sparktop looks for a config file in this order; the first that exists wins:
-config <path>(explicit override)$XDG_CONFIG_HOME/sparktop/config.yaml~/.config/sparktop/config.yaml← recommended~/.sparktop/config.yaml← legacy./config.yaml← dev convenience
Schema (see config.yaml.example):
interval: 1s # poll cadence; 1s is fine over a LAN/Tailscale
timeout: 10s # per-scrape budget (default: max(5s, 4×interval), ≤30s)
history: 60 # minimum points kept per chart (charts keep at least a screenful)
theme: tokyonight-night # optional; any built-in dado theme
nodes:
- name: spark-01
host: 192.168.1.101
vllm_port: 8000 # scrape vLLM /metrics: tok/s, queue depth, KV-cache, latency
- name: spark-02
host: 192.168.1.102
# node_port: 9100 # override if the exporters aren't on the defaults
# gpu_port: 9400sparktop is one binary with subcommands; the dashboard runs by default.
sparktop # dashboard (= sparktop dashboard)
sparktop deploy me@spark-01 ... # upload + bring up the exporter stack
sparktop teardown me@spark-01 ... # stop the stack (--purge removes ~/sparktop-exporters)
sparktop health spark-01 ... # probe TCP + /metrics on :9100 and :9400
sparktop version
sparktop help
deploy and teardown shell out to the system ssh, so your ~/.ssh/config,
agent, and known hosts all work as you'd expect.
The DGX Spark's GB10 superchip shares one ~128 GB unified LPDDR5X pool between the Grace CPU and the Blackwell GPU. That changes what monitoring matters: memory pressure is a node problem (an over-eager vLLM config can hard-lock the box, not just OOM the process), and thermal/power throttling on the compact chassis quietly eats your tok/s. sparktop is built around exactly those signals - unified memory %, GPU temp/power against the GB10 envelope, SM-clock throttle detection, and live vLLM serving stats - without standing up a Prometheus stack for a two-node cluster.
| Path | What it is |
|---|---|
cmd/sparktop/ |
the binary entrypoint (subcommand router) |
internal/cli/ |
subcommands: dashboard, deploy, teardown, health, version |
internal/config/ |
config file + flag parsing |
internal/metrics/ |
HTTP scrape + Prometheus-text parser + per-node snapshots/rates |
internal/ui/ |
the dado dashboard |
internal/exporters/docker-compose.yml |
node_exporter + dcgm-exporter stack - embedded in the binary |
Taskfile.yml |
task build, task test, … (needs Task) |
.goreleaser.yaml |
release builds, the .deb, and the Homebrew tap formula |
- DGX Spark is ARM64. The exporter images (
prom/node-exporter,nvcr.io/nvidia/k8s/dcgm-exporter) publishlinux/arm64, andtask build-armcross-compiles the TUI for the nodes too. - GPU metrics refresh.
dcgm-exportercollects on its own internal interval (coarse by default) - setDCGM_EXPORTER_INTERVAL=1000on the node if you want GPU metrics to actually move every second. dcgm-exporterGPU access. The compose file uses thedeploy.resources.reservations.devicessyntax; swap it forruntime: nvidiaif your Docker is set up the older way.- No GPU metrics? If
dcgm-exportercan't enumerate the GB10 on your DGX OS build, the dashboard shows "no GPUs reported" for that node and keeps working on host metrics. - Firewall. The host running
sparktopmust reachtcp/9100andtcp/9400(and your vLLM port, if configured) on each node. - Trade-offs vs. Prometheus + Grafana: same exporters, same metrics, but no long-term history (charts hold the last screenful of samples), no alerting, and it's local to your terminal rather than a shared web UI.
- A per-node drill-down view (full-screen, more panels)
- Threshold coloring + an alerts pane (GPU temp, disk full, node down)
- Optional on-disk history so charts survive a restart
- Mouse/scroll for picking a node
dado- the TUI toolkit sparktop is built on- paul-aviles/NVIDIA-DGX-Spark-Dashboard - the original inspiration
node_exporteranddcgm-exporter- the metrics sources