Nvidia's November seventeenth sparse-attention discussion adds another physical cost. Selecting only some keys doesn't eliminate the indexer's work over the long history. A related April GVR paper exploits overlap between adjacent decoding steps to perform exact top-k selection. Under its DeepSeek V-three-point-two, Blackwell and TensorRT-LLM setup, the operator averaged a one point eight eight times speedup, while the end-to-end time per output token improved by up to seven point five two percent. The smaller end-to-end gain reflects how much of decoding remained outside that accelerated operation.

Source 1 · Source 2