Meta Releases Muse Spark Open, 816B Open-Weight Model with 1M Token Context
Summary
Meta releases Muse Spark Open, an open-weight mixture-of-experts model with about 816 billion parameters, 42 billion active per token, and a 1 million token context window. It accepts text, images, audio and video and generates text, aimed at agentic coding. Meta calls it the most capable US open-weights model. The BF16 checkpoint is about 1.5 TB.
Key Points
- Meta's router selects 8 experts per token from 256 in each of the model's 62 layers.
- The 4-bit NVFP4 checkpoint runs on a single NVIDIA B200 node.
- LoRA fine-tuning for 500 steps on 64 H100 GPUs lifts mean token accuracy from 72% to 84%.