Skip to content

A short introduction to Mixture-of-Experts (MoE)

Panorama city view of Bryggen, Bergen, Norway

Photo by Michael Fousert

I’ve been discussing Mixture-of-Experts (MoE) a lot lately. The appeal is that it scales model capacity without paying the full compute cost for every token. Mixtral 8x7B brought this architecture to a lot of people.

The release of Mixtral 8x7B

Mixtral 8x7B was released in December 2023. The original post linked to Mistral’s announcement on X; its formal technical introduction followed on December 11. The architecture details below have been checked against that introduction and the later Mixtral paper.

Mixtral has 46.7 billion parameters in total, but it doesn’t run all of them for every token. It’s a sparse mixture-of-experts model: only a small subset of experts activates for each token, bringing the active parameter count per token down to ~12.9 billion. These are Mistral’s published counts; they do not imply that wall-clock speed always matches a dense 12.9B model.

Two-panel meme: a figure labeled 56b is stopped at a door, then revealed to be several smaller figures labeled 7b wearing one coat

The meme is loose about the architecture. Mixtral is not eight complete 7B models: the experts are feed-forward sub-networks, while attention and other components are shared. That is why the total is about 46.7B rather than 56B.

Mixtral 8x7B is a decoder-only model where each feedforward block has 8 distinct groups of parameters. At every layer, for every token, a router network chooses two of these experts to process the token, then takes a weighted sum of their outputs. Mixtral architecture.

This is top-2 routing (num_experts_per_tok: 2), which makes Mixtral a sparsely-gated MoE / top‑k routing model. Switch Transformer routes to a single expert per token (top‑1).

These selected architecture settings use the original inference-config naming. Their values correspond to the official model configuration, whose Transformers field names differ:

{
  "dim": 4096,
  "n_layers": 32,
  "head_dim": 128,
  "hidden_dim": 14336,
  "n_heads": 32,
  "n_kv_heads": 8,
  "norm_eps": 1e-5,
  "vocab_size": 32000,
  "moe": {
    "num_experts_per_tok": 2,
    "num_experts": 8
  }
}

For fixed-size experts, the expert computation per token scales with k, while expert-weight storage scales with the total number of experts. Attention, routing, and other shared operations still have their own costs. Sparse routing cuts expert compute per token, but it doesn’t remove the weight-storage cost, and routing can add overhead in distributed setups. Mixtral’s efficiency discussion.

Defining MoE

In the sparse MoE layers discussed here, computation is conditional: instead of running one feed-forward block for every token, the layer keeps several feed-forward blocks (experts) and uses a router to pick a small subset per token. MoE more broadly also includes mixtures that evaluate every expert; sparse routing is the part that avoids that work. Sparsely-gated MoE.

It can feel like an ensemble, but only the experts selected for a token actually run.

MoE ideas date back to early 1990s work on mixtures of local experts. Shazeer et al. (2017) applied sparse gating to large language models, building on that earlier work.

Core components

Mixture-of-experts (MoE) layer with only 2 experts selected and activated by the gating network

Image source: Shazeer et al., 2017

Experts are feed-forward sub-networks with separate weights, trained jointly with the router in Mixtral. Their name does not guarantee subject-matter specialization: the Mixtral paper found no obvious topic-based assignment pattern in its analysis, but did observe patterns related to syntax and nearby tokens. Mistral’s training description, routing analysis.

The router scores experts for each token, selects the top‑k experts, and assigns combination weights. In Mixtral, a linear layer produces the scores, and softmax normalizes the selected scores. Training often includes an auxiliary load-balancing loss to discourage concentrating work on a few experts; Switch Transformer is one documented example.

Replacing a dense FFN with an MoE does not itself require changing attention. Mixtral replaces every FFN with an MoE layer, though it also differs from Mistral 7B by using full attention over its 32K context rather than a sliding window. Its routing and weighted combination look roughly like this:

scores = router(h)              # one score per expert
ids = topk(scores, k)           # e.g., k = 2 for Mixtral
weights = softmax(scores[ids])  # normalize over selected experts
y = sum_i weights[i] * expert[ids[i]](h)

Costs and constraints

The Switch paper describes the capacity and communication trade-offs. Keeping every expert in accelerator memory is not mandatory: offloading research on Mixtral shows that weights can move between host and accelerator memory, at a transfer cost.

Where MoE fits

MoE can reduce FLOPs per token at a given quality level, and expert parallelism provides a way to distribute its weights across devices. It can also run efficiently on a single GPU with suitable kernels and enough memory; the Mixtral paper describes both arrangements. For local inference, the useful comparison is model quality against available memory, batching, and transfer costs. A smaller dense model can be simpler when the MoE weights do not fit, but a single device is not by itself a reason to rule out MoE.

Sources and further reading

  1. Fedus, W., et al. (2021). “Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity.” arXiv preprint arXiv:2101.03961. Link

  2. Zoph, B., et al. (2022). “Designing Effective Sparse Expert Models.” arXiv preprint arXiv:2202.08906v1. Link

  3. Du, N., et al. (2021). “GLaM: Efficient Scaling of Language Models with Mixture-of-Experts”. Link

  4. Shazeer, N., et al. (2017). “Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer.” arXiv preprint arXiv:1701.06538. Link

  5. Bengio, Y., et al. (2013). “Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation.” Link

  6. https://huggingface.co/blog/moe

  7. https://www.artfintel.com/p/papers-ive-read-this-week-mixture?nthPub=201

  8. https://www.artfintel.com/p/more-on-mixture-of-experts-models

  9. https://lilianweng.github.io/posts/2021-09-25-train-large/#mixture-of-experts-moe

  10. https://www.youtube.com/playlist?list=PLvtrkEledFjoTA9cYo_wX6aG2WT5RFBY9

  11. https://www.youtube.com/watch?v=ccBMRryxGog&t=1038s