Mixture-of-Experts Architectures: A 2026 Survey
How sparse mixture-of-experts designs are letting model capacity grow without a proportional rise in inference cost.
The core idea
Mixture-of-experts (MoE) architectures route each input through only a subset of the model's total parameters, letting total capacity scale up without every token paying the full inference cost of the entire model.
Recent refinements
Improved routing algorithms have reduced the training instability and load-balancing issues that made early MoE models harder to train reliably than dense architectures.
Why it matters
MoE is a major reason frontier models have been able to keep growing in effective capability without inference costs rising at the same rate.