NVIDIA Tests Show DSpark Speeds Nemotron on 64GB DGX Spark

NVIDIA published a local workflow Friday, Oct. 30, for running Nemotron-3.5 Lightning on its 64GB DGX Spark, with tests showing higher generation rates when DSpark was enabled.

Mean generation speed was higher with DSpark enabled in all three workloads. Median time to first token increased by 2.1 milliseconds or less in each test.

Workload Mean generation speed, DSpark off Mean generation speed, DSpark on Median time to first token, off/on

SpeedBench 1K 73.3 tokens per second 108.5 tokens per second 201.0 ms / 203.0 ms

SpeedBench 2K 72.8 tokens per second 105.3 tokens per second 329.5 ms / 331.6 ms

HumanEval prompts 73.4 tokens per second 126.9 tokens per second 98.1 ms / 98.7 ms

The tests used NVFP4, enabled thinking, batch size one, one concurrent streaming request and a 128-token output limit. DSpark used three draft tokens.

The command-line workflow installs TensorRT Edge-LLM and serves the model through a local OpenAI-compatible API at 127.0.0.1:8000. The first run downloads about 21GB of model files and builds inference engines; later runs reuse cached engines.

The example used TensorRT Edge-LLM 0.11.1 and TensorRT 10.16.1.11. Requirements include CUDA 13, TensorRT 10.16.1.11 or a newer version in the 10.x series, and Python 3.10 to 3.12.

Peak total system memory use during inference with DSpark was 27.9GB, including the operating system. The test ran on DGX OS 7.6.0, based on Ubuntu 24.04.4 LTS.

TensorRT Edge-LLM is a lightweight C++ runtime for embedded deployments. NVIDIA designed the inference framework to extend across DGX Spark and its Jetson Thor, DRIVE AGX Thor and IGX Thor platforms. The workflow applies to the 64GB DGX Spark NVIDIA made available through manufacturer partners on Oct. 28.