DeepSeek-AI Publishes DeepInfer Inference Framework Under Apache 2.0
Summary
DeepSeek-AI published DeepInfer, the inference framework it says powers its production services, under Apache 2.0. It serves DeepSeek-V4.1-Flash on Nvidia SM100 GPUs and Huawei Ascend 950 NPUs. The release omits a KV cache system and prefill/decode disaggregation, which the repository says a complete service requires.
Key Points
- DeepSeek-AI says DeepInfer was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM.
- Deterministic sampling and batch-invariant inference run on Nvidia hardware only, gated behind an environment flag the repository names DEEPINFER_BATCH_INVARIANT=1.
- The repository lists 16.9k stars, 1.7k forks and four commits, the last on Oct 21 bumping DeepEP and dropping an SSH to HTTPS rewrite.