NVIDIA Engineers Claim 2.1x to 2.4x Faster Top-K by Reusing Prior Decode Selection
Summary
NVIDIA engineers recommend eight design rules for sparse attention on its GPUs, including keeping group size times speculative draft length within one Tensor Core tile of about 200 rows. They report that reusing the previous decode step's Top-k selection is 2.1x to 2.4x faster than standard radix select.
Key Points
- An FP16 epilogue widens the indexer Tensor Core tile to 256 rows, giving 1.31x at G=64, NVIDIA's authors report.
- Guideline 3 calls for KH = 1 with a cross-head reduction over query heads, so one logit per key is written.
- The indexer and Top-k take 47% of a DeepSeek V3.2 prefill layer at 128K tokens, NVIDIA reports.