NVIDIA Engineers Claim 2.1x to 2.4x Faster Top-K by Reusing Prior Decode Selection
NVIDIA engineers recommend eight design rules for sparse attention on its GPUs, including keeping group size times speculative draft length within one Tensor Core tile of about 200 rows. They report that reusing the previous decode step's Top-k selection is 2.1x to 2.4x faster than standard radix select.