# NVIDIA Finds Key Scoring and Selection Took 47% of Layer Prefill Time

By Simon Yoon

Canonical URL: https://www.tokenpost.com/news/technology/60107
Published: 2026-11-18T04:03:48.000Z
Updated: 2026-11-18T04:03:48.000Z
Section: Technology

> The DeepSeek V3.2 example used a 128K-token input. Separate tests found that more indexer key heads and smaller selection blocks increased component runtimes.

NVIDIA found that scoring and selecting what an AI model attends to can consume a large share of processing time, complicating efforts to run long prompts efficiently on GPUs.

The company's Nov. 17 guidance used a DeepSeek V3.2 example with 128K input tokens. In that example, the indexer and Top-k selection accounted for 47% of per-layer prefill time. Prefill is the stage when a model processes an input before generating a response.

Sparse attention limits which keys each query attends to in the main attention calculation. The indexer scores possible keys across the context, while Top-k chooses which enter that calculation. Scoring the context remains GPU work even when the main calculation uses fewer keys.

The design choices matter for applications that process long documents, large codebases and agentic workloads. Inference includes both prefill and decode, the stage when the model generates its answer. NVIDIA examined how those choices affect parts of each stage.

In one component test, increasing the number of indexer key heads from one to eight made the indexer run 7.6 times as long during decode. That measurement covers indexer runtime, a specific part of generating a response.

A separate test selected 2,048 keys for each query and changed how they were grouped. Halving the size of contiguous selection blocks nearly doubled core-attention runtime in both prefill and decode. The number of selected keys stayed fixed in that comparison, making block size the design choice under examination.

Contiguous blocks allow GPUs to fetch and reuse data efficiently. The block-size result shows that the number of selected keys alone does not determine the measured runtime. How those keys are grouped also affects the work involved in core attention.

The 47% figure measures one layer's prefill time in the 128K-token DeepSeek V3.2 example. The 7.6-times increase and the block-size result come from separate component tests, each measuring a different part of inference.

NVIDIA's guidance recommends scoring fewer keys, using one indexer key head shared across query heads and selecting larger contiguous blocks. Those choices address the amount of scoring work and how efficiently GPUs retrieve the selected data.


