Mixture of Experts (MoE): Sparse Routing & Cost Savings

Mixture of experts (MoE) is a neural network design where, instead of one large network processing every input, the network is split into many smaller sub-networks called “experts,” and a separate small “router” network decides, for each input, which expert or small subset of experts should handle it.

How it works: experts and the router

In a standard neural network, every layer processes every piece of data. MoE changes this structure. Each expert is typically a smaller neural network, often the feed-forward part of a transformer layer. These experts specialize in patterns within the data. This specialization emerges from the training process rather than being assigned by a person.

The router looks at each input. In a large language model, this usually means each token. The router outputs which one or few experts from the total set should process that specific input. This routing mechanism allows the model to scale its capacity without increasing the computation required for every single token.

Why sparse activation saves compute

The core benefit of this architecture is sparse activation. Because only a small subset of experts is activated for any given input, an MoE model can have a very large total parameter count. Yet, the actual amount of computation used to process each input stays much closer to that of a much smaller dense model.

This contrasts with a dense model where every parameter is used for every input. In an MoE setup, the compute cost per token remains low because the router activates only the necessary experts. This efficiency is what makes large-scale MoE models viable for practical deployment.

Origins of the idea

The core idea goes back to the 1991 paper “Adaptive Mixtures of Local Experts” by Robert Jacobs, Michael Jordan, Steven Nowlan, and Geoffrey Hinton. That paper proposed training multiple expert networks together with a gating network that learns to route inputs to the appropriate expert. This foundational work established the theoretical basis for modern sparse routing in artificial neural networks.

Modern example: Mixtral

A prominent modern example is Mixtral 8x7B. Mistral AI released this openly released mixture-of-experts language model in December 2023. It features 8 experts per layer. The model routes each token to 2 of those experts at a time.

The company presented Mixtral as matching or beating larger dense models on several benchmarks while using less compute per token at inference time. This demonstrates how sparse routing can deliver high performance without the linear cost increase seen in traditional scaling methods.

MoE vs a dense model

When comparing an MoE-based LLM to a traditional dense architecture, the differences are significant. A dense model of the same total parameter count as an MoE model would use all of its parameters on every input. This generally makes it slower and more expensive to run per token than the sparsely activated MoE model.

However, there is a nuance in memory usage. The MoE model still needs enough GPU memory to hold all of its experts. The model weights are stored in memory regardless of whether they are actively processing a token. This distinction is crucial for understanding hardware requirements.

Tradeoffs

MoE models introduce complexity. They are more complex to train than dense models. The router has to learn to distribute inputs sensibly across experts. If the routing is not balanced, training can become unstable. A common issue is when inputs collapse onto only a few experts instead of using the full set. This leads to underutilized experts and reduced model capacity.

Serving an MoE model also has specific hardware implications. Serving an MoE model still requires enough memory to store every expert. Even though only a few are used per input, the memory footprint remains large. Consequently, the memory savings compared to a dense model of similar capability are smaller than the compute savings. You save on FLOPs (floating-point operations), but not on RAM or VRAM capacity.

FAQ

What is a moe model?

An MoE model is a neural network that divides its parameters into multiple “expert” sub-networks. A router directs each input to only a few of these experts, allowing the model to scale in size while keeping inference costs low.

How does sparse mixture of experts work?

Sparse activation means that for any single input, only a small fraction of the total model parameters are active. The router selects the most relevant experts for that specific input, skipping the rest.

Is Mixtral an MoE model?

Yes. Mixtral 8x7B is a mixture-of-experts language model released by Mistral AI. It uses 8 experts per layer and routes tokens to two experts at a time.