NVIDIA Reports 42.5% Throughput Gain for Ling-3.0-flash-fp4 on DGX Spark

NVIDIA said three software changes lifted decode throughput for Inclusion AI's Ling-3.0-flash-fp4 by 42.5% on DGX Spark in a test covering all 1,319 GSM8K questions, while mean latency fell as well.

Throughput increased to 53.89 tokens per second from 37.81, and mean latency dropped to 11.49 seconds from 15.21. The end-to-end test ran at batch size one.

The model has 124 billion total parameters, with 5.1 billion active for each token. NVIDIA used a Humming MXFP4 mixture-of-experts backend, online FP8 quantization of the language-model head and NEXTN multi-token-prediction speculative decoding. The first two changes reduce the cost of each decode step; the third aims to reduce how many steps are needed.

GSM8K scores were 96.89% for the baseline and 96.82% with all three changes. The optimized run's mean MTP acceptance length was 2.68 tokens per round, compared with 3.90 in a separate synthetic test.

That batch-size-one synthetic test used 32,000 input tokens and generated 2,000 output tokens. Throughput reached 102.88 tokens per second, 185.1% above baseline. NVIDIA cautioned that the result does not establish performance on other workloads, saying repetitive outputs and greedy decoding made the setup overstate speculative-decoding benefits.

The GSM8K run used five-shot prompting, a temperature of 0.6, top-p of 0.95 and top-k of 29. Chain-of-thought was enabled, with a 32,768-token maximum. GSM8K outputs had a median length of about 310 tokens, compared with the synthetic test's fixed 2,000-token generation.

The combined FP8 and NEXTN setup also has a model compatibility requirement: the draft model must share the full language-model-head module with the target model. NVIDIA said the framework can otherwise fall back to a configuration with an acceptance length of 1.00 and throughput below baseline.

The benchmark follows an Oct. 22 update to NVIDIA's local AI systems, including DGX Spark.