Nvidia's November seventeenth sparse-attention discussion adds another physical cost. Selecting only some keys doesn't eliminate the indexer's work over the long history. A related April GVR paper exploits overlap between adjacent decoding steps to perform exact top-k selection. Under its DeepSeek V-three-point-two, Blackwell and TensorRT-LLM setup, the operator averaged a one point eight eight times speedup, while the end-to-end time per output token improved by up to seven point five two percent. The smaller end-to-end gain reflects how much of decoding remained outside that accelerated operation.
An accelerated operator is only part of decoding
A faster top-k operation yields a smaller end-to-end decoding improvement.
2 sources · 1 min read
Link to noteOriginally published 20 November 2026. Reporting, forecasts, prices, odds and event availability reflect the original edition.