Inferact Open-Sources TPU v7 Megakernels, Claims Kimi K3 Tops 700 Tokens/s

Inferact open-sources TPU v7 megakernels and claims its Kimi K3 implementation surpasses 700 tokens per second with speculative decoding on 16 TPU v7 chips, using one Pallas kernel to fuse all 92 MoE layers.

• Inferact releases open-source tpu-megakernels for TPU v7, claiming its Kimi K3 implementation exceeds 700 tokens/s with speculative decoding versus 452 tokens/s on 16 GB200 GPUs.
• The single Pallas megakernel combines Kimi K3's 92 MoE layers and uses asynchronous HBM-to-VMEM prefetching across layer boundaries on 16 TPU v7 chips.
• The Kimi K3 kernel compiles from scratch in under 90 seconds, versus 30-plus minutes that large TPU models can require with hundreds or thousands of XLA operations.