brain

dgx-station-frontier

Properties12
descriptionReceipts-first serving recipes and measurements for two NVIDIA DGX Station GB300s
statusseedling
typesource
created2026-10-12
updated2026-10-12
tagsoss
aliases
sourcehttps://github.com/RonanLabsAI/dgx-station-frontier
languagePython
licenseApache-2.0
stars4
public

10 min read

dgx-station-frontier

Receipts-first serving recipes and measurements for a pair of NVIDIA DGX Station GB300 desktops, from RonanLabs. Every number in this repository is one row in results.jsonl, and every row points at a scrubbed log excerpt under logs/. tools/validate.py checks both directions: each row against its log, and each performance number in the READMEs against results.jsonl.

Six recipes and a microbenchmark atlas (the Nemotron and Kimi K3 recipes were added after the first release).

FolderWhat it is
recipes/dsv41-flash-sidecar/DeepSeek-V4.1-Flash on one Station, with the RTX PRO 6000 serving the cold experts (original-el8’s sidecar design on public b12x). Also run as two independent copies, one per Station (DP2). Plus prefill-biased expert slots (pre-registered): +7% 16K prefill against our own same-session recipe, decode unchanged, long-context accuracy non-inferior
recipes/dsv4-pro-two-stations/DeepSeek-V4-Pro-0813 (1.6T parameters) across both Stations: E1 stock, E2 hot/cold expert split, E3 with an RTX PRO 6000 warm-expert tier on each Station. Plus the bundled DSpark drafter: hung unfixed, then made servable by a rank-symmetric guard and skip-draft (20-min streaming soak passed)
recipes/nemotron-ultra/NVIDIA Nemotron 3 Ultra 550B-A55B NVFP4 (general and IOI competitive-coding checkpoints) on one Station with routed experts in Grace memory, and on two Stations with every weight in HBM. Stock vLLM, no patches
recipes/kimi-k3/Moonshot Kimi K3 at 3 bits (vellum W3A16) across both Stations in vLLM, with most routed experts in Grace memory (load hook and four load fixes included); and the 1.6-bit GGUF in llama.cpp on one Station, two Stations and four GPUs, with our C2C zero-copy patch and a CUDA 13.2 miscompile warning
recipes/mimo-v2.6-pro/Xiaomi MiMo-V2.6-Pro: original-el8’s three-tier design on one Station with a W4A16 sidecar (HBM + RTX PRO 6000 + Grace), and six tiers on two Stations (Grace empty) with a rank-0 routing broadcast that keeps the TP ranks’ expert choices identical. Receipts and notes; the patched kernel and hooks are not published yet
recipes/glm53flash-tensorfold-gb300/GLM-5.3-Flash (EXL3) on one GB300 with Ash Hart’s TensorFold engine, from Mia’s DGX Spark recipe plus a two-line single-GPU patch and two tuning patches.
recipes/fabric-data-direct/The cross-Station fabric: Data Direct inside containers, a per-message-size NCCL tuner, the lossless traffic class, and what small-message latency does and does not allow
atlas/microbench/C2C, HBM, Grace STREAM, RTX PRO 6000 to Grace, CPU vs GPU expert compute, GEMM power
articles/Plain-English write-ups of the results above. Where should your experts live? (Grace vs HBM vs RTX PRO 6000); What does an expert sidecar actually do?; One Station with a sidecar vs two Stations without; Running a 1.6T model on two desktops (DeepSeek-V4-Pro across two Stations); Should you add an RTX PRO 6000 to your DGX Station?; MiMo-V2.6-Pro on two desktops, with nothing left in slow memory; Speculative decoding: same server, twice as fast or half as fast

Hardware

Two DGX Station GB300 desktops. Each has:

  • one GB300 GPU (256,703 MiB of HBM3e visible, power.limit 1,300 W) joined to a 72-core Grace CPU with about 494 GiB of LPDDR5X over NVLink-C2C;
  • one RTX PRO 6000 Blackwell Max-Q (97,887 MiB, power.limit 300 W) on PCIe;
  • one ConnectX-8 SuperNIC with two 400G ports.

All cross-Station results use one 400G DAC between the two ConnectX-8s (RoCEv2, no switch). The second port is not cabled yet. Software for every row: driver 595.91.07 (except the Kimi K3 vLLM rows: 595.99.02), host CUDA 13.2, kernel 7.0.0-1019-nvidia-64k, Ubuntu 24.04.4. Engines run in containers pinned by digest; each row of results.jsonl names its image digest and versions.

Headline results

Shape unless noted: 8,192 random-token prompt, 1,024 forced output tokens, temperature 0 (the shape catid published for Station-pair results, so the numbers compare). “C” is concurrency; aggregate = output tokens per second across all streams.

ResultOursReceiptBest public comparison
DS-V4.1-Flash, one Station + its RTX PRO 6000 as cold-expert sidecar, C16 aggregate1,856 tok/slog1,628.6 tok/s, same design on one Station (el8); 1,677.7 tok/s on two Stations (catid)
DS-V4.1-Flash DP2: two independent copies, one per Station, C32 / C64 aggregate (sum of both)4,548 / 5,774 tok/sleft, rightone model across two Stations: 2,117 / 3,401.6 tok/s (catid TP2)
DS-V4.1-Flash, C1 per user (each DP2 copy, mean of three warm reps)445 / 452 tok/sleft, right
DS-V4-Pro-0813 across both Stations, stock vLLM (E1): C1 / C16 aggregate / GSM8K-20037.8 tok/s / 109.5 tok/s / 98.0 %logon one Station: @antirez, Q2 quant in DwarfStar (post); ours is released precision in vLLM
DS-V4-Pro with RTX PRO 6000 warm tiers (E3-A16): C16 aggregate, random ids / private real text; GSM8K-200164.6 / 230.8 tok/s; 97.5 %log
Nemotron-3-Ultra NVFP4: one Station (TP1, experts partly in Grace) C1 / C32 aggregate; two Stations (TP2+EP2, all in HBM, IOI coding checkpoint of the same architecture) C1 / C32 aggregate39.7 / 155.2 tok/s; 99.1 / 889.1 tok/sone Station, two Stationsnone found on DGX Station
Kimi-K3 with int3 experts (vellum W3A16) across both Stations, vLLM TP2+EP2, 408 GiB per Station of routed experts in Grace: C1 / C16 aggregate (upper bound, prefix-cache hits) / GSM8K-20018.1 tok/s / 52.17 tok/s / 98.0 %lognone found on DGX Station; on sixteen DGX Sparks with speculative decoding and a different prompt shape: 29.81 tok/s C1 (ciprianveg)
Fused GLM-5.3 TP2+EP2 over the rail: 64K -token prefill with Data Direct + tuner3 + traffic class 106 vs stock11,173 vs 8,033 tok/s (+39 %)log
Smallest cross-Station all-reduce on the engine path (14 KB, CUDA graph)31.67 µslog
GB300 reading pinned Grace memory over C2C (GPU kernel, zero-copy)352.8 GB/slog

What the comparisons mean:

  • One Station beats the published two-Station C16 figure with the same model, because the RTX PRO 6000 takes the cold experts. It is not a like-for-like topology comparison, and we say so wherever we quote it.
  • DP2 is two independent servers. One prompt never spans Stations; a load balancer splits requests. It is the right comparison only for aggregate throughput. Two Stations together still beat the published one-model TP2 run on aggregate (2.15x at C32 and 1.7x at C64), with different hardware in use (both RTX PRO 6000s).
  • The DS-V4-Pro real-text numbers use private prompts and a hot/warm expert map fitted to private traffic. Neither is published, so those numbers are not reproducible from this repo; the random-id numbers are.

Result pages:

Methodology

Benchmark shape. Decode: vllm bench serve (or SGLang’s bench_serving) with random token ids, 8,192 in, 1,024 forced out (--ignore-eos), temperature 0. DS-V4.1 rows report vLLM’s output token throughput; DS-V4-Pro rows report decode throughput = C x 1000 / mean TPOT. Each recipe README says which.

Bench hygiene. In October 2026 we audited our own receipts and found prefix-cache reuse inflating some of them. Since then every run follows three rules:

  1. A fresh seed for every invocation: per run, per concurrency, per repetition. vLLM’s and SGLang’s random datasets are prefix-stable, so with one seed the C8 prompt set contains the C1 prompts and C16 contains C8.
  2. The warm-up uses its own seed. Never use vLLM’s --num-warmups: it replays the first measured prompt, which then hits the cache when it is measured.
  3. Measure the prefix-cache hit rate around every measured run from the server’s /metrics counters and reject any random-token run above 2%.

The DS-V4-Pro E2, E3 and speculative rows follow all three rules (every reported run: zero measured hits; one contaminated repetition was discarded and is visible in its log). The Nemotron rows run with prefix caching off on the server (zero hits by construction, still checked per run). The DS-V4.1 and fabric rows were measured before the rules and were audited instead: the decode figures are clean (the only cache hits were warm-up copies hitting each other), except the C1 means, which reuse one prompt set by design and read a few percent high. Each row’s notes field says which applies.

Audit corrections, stated plainly.

  • Withdrawn: “DS-V4.1 64K-token prefill at C1 probably beats el8’s 44.5K tok/s.” Our 49.8K tok/s included one cached prompt out of four, worth roughly a fifth of the figure. The clean rerun (2026-10-09, fresh seeds, zero prefix hits) measured 38.6K tok/s, which does not beat it (and the prompt shapes differed anyway).
  • Withdrawn: “MiMo-V2.6-Pro on one Station beats J-M’s 63.0 tok/s at C16.” Our 65.38 tok/s ran with half of its prompt tokens served from the prefix cache. The clean rerun (2026-10-09) measured 53.26 tok/s output (59.67 tok/s decode), below 63.0: the beat is refuted. That recipe is not in this release.
  • Corrected: an early DS-V4.1 C1 gap between two expert maps (one run each) was speculative-decoding acceptance noise. Four alternating boots with three repetitions each showed no difference. We now quote C1 only as a mean of at least three repetitions.
  • Corrected: an early “one-Station DS-V4.1 is below el8” reading came from benchmarking right after a cold JIT compile. Warm, it is above.
  • Restated bar: our DS-V4-Pro numerics gate first asked for “under 1% top-1 flips vs stock”. Stock vLLM on this path flips 3.22 % against itself, so we judge the excess over that floor instead and report both.

Quality gates. Throughput is reported only next to a quality receipt: GSM8K (first 100 or 200 test rows) on the same boot, a per-layer cosine check of any sidecar against the stock kernel on real prompts, and for DS-V4-Pro a teacher-forced top-1 flip test against the stock server.

Power. Rows record power.limit and the highest power.draw sampled in the run window. Neither GPU came near its limit in any recipe.

Credits

  • original-el8 (Jason Cook), dgx-station-gb300-research (Apache-2.0): the graph-safe shared-memory expert sidecar, the hook design and the DS-V4.1 expert map that our DS-V4.1 recipe runs unchanged, and that our DS-V4-Pro E3 code is ported from (headers and NOTICE kept).
  • b12x / Local Inference Lab, b12x (Apache-2.0): the MoE kernels the sidecar runs on the RTX PRO 6000. E3-A16 uses our fix for a W4A16 bug on DS-V4-Pro shapes (offered upstream separately).
  • catid (Christopher Taylor), dgx_station_benchmarks: the two-Station reference numbers and benchmark shape we compare against, and the first public description of the Data Direct container overlay. That repository has no licence, so nothing from it is copied here; our overlay helper was written from our own notes.
  • J-M-Recipes (James Meadlock), recipes (MIT): the pin-hot-experts hook our DS-V4-Pro E2 hook is ported from (MIT notice kept), and the decode harness used for the GLM fabric A/B (not vendored).
  • Moonshot AI (Kimi K3, Kimi K3 License), Vellum (the 3-bit Kimi-K3-W3A16-g64 quantisation) and unsloth (the UD-IQ1_M GGUF) for the Kimi K3 weights; no weights are included. ciprianveg, gb10-vllm: the sixteen-Spark Kimi K3 reference numbers (no licence; nothing copied).
  • llama.cpp / ggml (MIT): our C2C zero-copy patch is against it, and the CUDA 13.2 reproducer follows one of its kernels (MIT notice kept in the file).
  • vLLM, SGLang, FlashInfer, NCCL and nccl-tests, perftest: used as released. Full third-party list: NOTICE.

Layout and validation

results.jsonl              one row per number: metric, value, unit, config, date (PT), versions, image digest, log
logs/<recipe>/*.log        scrubbed excerpts of the raw logs (hostnames, IPs, users, GPU UUIDs replaced)
recipes/<name>/            README, launchers, hooks, tools for that recipe
atlas/microbench/          microbenchmark sources and README
tools/validate.py          results.jsonl <-> logs <-> README numbers
tools/scrub_check.sh       fails on internal hostnames, IPs, user names, paths, tokens

python3 tools/validate.py && bash tools/scrub_check.sh

Licence

Apache License 2.0 (LICENSE). Third-party origins and notices: NOTICE. Contributions: CONTRIBUTING.md.

Graph View