#184 - Parameterize the bench suite for Spark/IGX and add IGX Orin performance report#185
Draft
dleshchev wants to merge 4 commits into
Draft
#184 - Parameterize the bench suite for Spark/IGX and add IGX Orin performance report#185dleshchev wants to merge 4 commits into
dleshchev wants to merge 4 commits into
Conversation
…ofiles The DGX Spark sweep configs/scripts were hardcoded for Spark (GB10), so running the same suite on the IGX Orin devkit (discrete RTX 6000 Ada + ConnectX-7) meant editing files in place. Make the suite select a target platform with BENCH_PLATFORM (default spark) instead. - examples/bench_platform.sh sources examples/bench_platform_<P>.env and fills @var@ placeholders in the sweep config templates. - bench_platform_{spark,igx}.env carry the per-platform values: GPU memory kind, num_bufs, core map, DPDK port BDFs, wire-loopback netdevs/RDMA devices, and the RoCE flow-control depth cap. - The three *_spark* sweep configs become platform-neutral templates; the run_spark_* scripts and setup_spark_wire_loopback_netns.sh source the helper and fill a concrete base before any generator/PyYAML parses it. The key per-platform difference is GPU memory placement: Spark's unified memory uses kind: host_pinned, while IGX's discrete GPU uses kind: device for real GPUDirect (DPDK raw ~96 vs ~74 Gb/s, RoCE large-message ~96 vs ~38 Gb/s -- the NIC reads VRAM over the GPU<->NIC PXB path instead of CPU DDR over PCIe). Spark-filled output is byte-identical to the previous hardcoded config; IGX validated end-to-end. Also fixed two run_spark_bench.sh issues surfaced on IGX: the socket per-pair port substitution required quotes that gen_spark_netns_config.py (PyYAML) strips, so every pair collided on the base port; and the RoCE client config used socket_config.remote_addr, which the current schema rejects (peer now lives in rdma_bench_client.server_address). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Denis Leshchev <dleshchev@nvidia.com>
Add docs/benchmarks/performance-igx-orin.md, the IGX Orin devkit counterpart to the DGX Spark performance report, produced by the same sweep scripts with BENCH_PLATFORM=igx. Single-rep (REPEATS=1) C++-loopback numbers for DPDK raw (kind: device GPUDirect), RoCE SEND, TCP, and UDP, with an "Single-rep figures" callout in lieu of error bars. Headline: the offload-heavy bulk paths (DPDK raw, RoCE >=64 KB) hit the same ~96-98 Gb/s NIC/PCIe ceiling as Spark, while the CPU-bound paths trail Spark in proportion to the per-core CPU gap -- IGX's 1.97 GHz Cortex-A78AE cores vs the 3.9 GHz Cortex-X925 cluster the Spark bench pins onto. The single-pair 1 MiB TCP ratio (17.8 vs 31.6 = 1.78x) tracks the clock ratio, confirming the wire is not the limit. - scripts/plot_mq_payload_sweep.py: parameterize the chart title (optional 3rd arg) so it is not hardcoded "DGX Spark"; render docs/images/igx-mq-payload-sweep.svg. - Wire the page into mkdocs.yml nav, the README Documentation table, and the AGENTS.md docs layout list; add a Spark->IGX cross-link on the Spark page. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Denis Leshchev <dleshchev@nvidia.com>
Re-ran the full IGX Orin suite with REPEATS=3 (BENCH_PLATFORM=igx) and replaced the single-rep figures with 3-rep means. The numbers confirm the single-rep draft almost exactly (run-to-run spread <=0.4 Gb/s on DPDK/RoCE, <=0.3 on sockets), so the narrative is unchanged. Notable corrections from the cleaner data: - RoCE server RX core is ~4% busy (idle), not ~76% -- the expected RoCE RC signature (HCA writes straight to memory), matching Spark. The single-rep draft mis-attributed a busy worker core. - DPDK CPU-utilization table replaced with an honest note: per-core busy% is unreliable on the nohz_full/rcu_nocbs isolated cores (tickless /proc/stat accounting under-counts and shifts run-to-run at identical throughput), so we don't publish precise per-core figures -- the throughput-vs-Spark gap is the real CPU-bound evidence. - Regenerated docs/images/igx-mq-payload-sweep.svg from rep-averaged data. Methodology callout updated to REPEATS=3; Reproduce section exports REPEATS=3. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Denis Leshchev <dleshchev@nvidia.com>
…t results The IGX socket pairs were pinned to the isolated cores (9 10 11 0) that DPDK/RoCE use for poll-mode busy-spin. That tuning is wrong for kernel sockets: the nohz_full/rcu_nocbs cores defer timers/RCU and starve the syscall path, and core 0 is the single sink for all NIC RX softirqs. Re-pin SOCKET_PAIR_CORES to the non-isolated cores (1 2 3 4) in the irqaffinity=0-8 pool and re-run TCP+UDP at REPEATS=3. Effect: small-message TCP now scales ~linearly (1000 B: 4.2/8.5/16.7 over 1/2/4 pairs vs 3.6/7.1/12.1 before) and UDP improves across the board (8 KB 4-pair 10.5 vs 9.3 Gb/s). Large-message TCP is essentially unchanged because it is receive-side bound, not app-core bound. Report update: refreshed the TCP/UDP tables and added the measured root cause for the large-message TCP ceiling -- all NIC RX softirq serializes on a single core (cpu0 runs ~70% in softirq, every other core sees no NET_RX), so the four flows' copy-heavy receive paths share one softirq core. RSS/RPS is not spreading receive work; lifting it would need RX-softirq spreading, not more app cores. Documented the socket-vs-poll-mode core split in the System-under-test table. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Denis Leshchev <dleshchev@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Closes #184 (IGX Orin C++ benchmarking) and makes the DGX Spark performance suite portable.
Two parts:
Parameterize the bench suite for Spark and IGX via platform profiles. The
*_spark*sweep configs andrun_spark_*/setup_spark_wire_loopback_netns.shscripts were hardcoded for DGX Spark (GB10). They now select a target withBENCH_PLATFORM(defaultspark, so existing Spark usage is unchanged):examples/bench_platform.shsourcesexamples/bench_platform_<P>.envand fills@VAR@placeholders in the config templates.bench_platform_{spark,igx}.envcarry the per-platform values: GPU memory kind,num_bufs, core map, DPDK port BDFs, wire-loopback netdevs/RDMA devices, RoCE flow-control depth cap.socket_config.remote_addr, which the current schema rejects.Add the IGX Orin performance report (
docs/benchmarks/performance-igx-orin.md), the counterpart to the DGX Spark page, produced by the same scripts withBENCH_PLATFORM=igx.Key finding
The offload-heavy bulk paths (DPDK raw, RoCE ≥64 KB) hit the same ~96–98 Gb/s NIC/PCIe ceiling on both platforms. The CPU-bound paths (kernel sockets, small-message RoCE, small-packet DPDK) trail Spark in proportion to the per-core CPU gap — IGX's 1.97 GHz Cortex-A78AE cores vs the 3.9 GHz Cortex-X925 cluster the Spark bench pins onto. The single-pair 1 MiB TCP ratio (17.8 vs 31.6 Gb/s = 1.78×) tracks the clock ratio, confirming the wire is not the limit. On the discrete RTX 6000 Ada,
kind: devicegives real GPUDirect (NIC DMAs into VRAM).Draft status / follow-ups
REPEATS=1) — the report carries a callout in place of error bars. AREPEATS=3re-run is planned before un-drafting, which will also clean up the single-queue CPU-utilization sampling.*_spark*files to platform-neutral names (touches the CI doc-ref check + AGENTS table), and a Spark hardware re-validation pass.Docs sync
mkdocs.ymlnav, README Documentation table, AGENTS.md docs layout, and a Spark→IGX cross-link all updated.scripts/check_doc_refs.pypasses.🤖 Generated with Claude Code