Mixture of Experts (MoE) Study Notes

I. Preface

With the rapid development of large language models and the broader deep learning field, MoE (Mixture of Experts) has gradually become a research hotspot thanks to its conditional computation characteristics and strong scalability. This article organizes the basic principles of MoE models, their overall architectural design, and key core techniques. It also references a few open-source examples to briefly describe typical implementation approaches and common application scenarios.

II. What is MoE?

Before diving deeper into MoE, let’s first review the classic Transformer architecture. In a standard Transformer, each Encoder/Decoder layer contains multi-head attention (Multi-Head Attention) and a fully connected feed-forward network (Feed-Forward Network, often abbreviated as FNN). This FNN applies the same linear transformation and nonlinear mapping to all input tokens, resulting in dense computation and high resource consumption.

The Mixture of Experts (MoE) technique introduces an “expert” mechanism at the position of the FNN: it splits a large feed-forward network into multiple sub-networks (experts), and uses a gating network (Router) to dynamically route tokens. Concretely, the gating network computes a score for each expert based on a token’s features (e.g., embeddings or attention context), and activates only the top K experts to participate in computation while the rest remain idle. In this way, a model can scale to hundreds of billions of parameters, yet during each forward pass only a small portion of parameters are actually activated (Activated Parameters)—typically around 5% to 15%. This greatly reduces compute overhead and GPU memory usage.

• The essential difference between Transformer and MoE
  • Dense computation: In a standard Transformer, the FNN treats all parameters equally.
  • Conditional computation: An MoE model computes only with a subset of experts, achieving sparse activation.
• Why design it this way? Key motivations include:
  • Scalability: When the budget allows, you can increase the number of experts to expand model capacity.
  • Efficiency: Fewer activated parameters can immediately reduce FLOPs and GPU memory usage.
  • Specialization: Different experts can “specialize” in different feature types or tasks, improving overall performance.

Imagine a library (like a Transformer) that holds tens of thousands of books (parameters). To find some information, you would have to flip through all the pages. An MoE model, by contrast, is like a librarian (the gating network) who selects only the most relevant books (experts) based on your question—saving time and improving efficiency.

III. Core components of MoE

MoE mainly relies on two modules:

1. Experts
   • Each expert is an independent feed-forward sub-layer (usually two linear transformations with an activation function). You can think of this as specialized training over a subset of the model’s parameter space (see Outrageously Large Neural Networks: The Sparsely‑Gated Mixture‑of‑Experts Layer).
   • Experts can be partitioned using different strategies, such as Balanced K‑Means clustering, Co‑activation Graph partitioning, or Gradient-aware partitioning, to ensure each expert has an optimal parameter configuration in its domain.
2. Router (Gating Network)
   • The input $x$ is linearly projected to expert scores: H(x)=xW\_g, and then a learnable noise term is added for smoothing: H'(x)=H(x)+SoftPlus(xW\_n)⊙𝒩(0,1) (this is called Noisy Gating).
   • It uses a Top‑K selection strategy: keep the K experts with the highest scores, normalize, and output G(x)=Softmax(KeepTopK(H'(x),K)). Then it routes the input to those selected experts.
   • An auxiliary load loss is also used: L\_aux=λ·CV(n\_1,…,n\_E), to minimize the variance of the number of tokens assigned to each expert and avoid overloading “popular experts.”

IV. Why is MoE so efficient?

1. Reducing FLOPs via conditional computation

In a standard Transformer, the FNN must perform the same computation for all N expert sub-layers and all input tokens, yielding complexity on the order of O(N·d^2). MoE activates only K experts, reducing the complexity to O(K·d^2). When K is much smaller than N, compute can drop by nearly a factor of N/K.

2. Parameter over-provisioning and representation power

Very large models often require massive numbers of parameters to fit complex patterns, but dense scaling leads to compute bottlenecks. With sparse routing, MoE allows a global parameter count several times larger than the activated parameter count (E·d^2) during training, improving the model’s ability to capture diverse patterns.

3. Specialist division of labor and distributed parallelism

Each expert only needs to learn features in a subspace, enabling functional division of labor. With expert parallelism (Expert Parallelism), different experts can be deployed on different devices, reducing synchronization overhead in the backbone network. Combined with pipeline or tensor parallelism, MoE can achieve higher throughput and lower latency (see GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding).

4. Practical evidence

• The GLaM model, with 1.2 trillion parameters, activates only 8% of them and outperforms GPT‑3 (175B) on 29 NLP tasks—highlighting the advantage of conditional computation.
• Switch Transformer uses a Top‑1 routing strategy, nearly doubling throughput while maintaining task performance.

V. Technical details of MoE

Traditional large neural networks (such as the common Transformer) behave like an “all-rounder”: every time they process data, they have to use all parameters for computation, which is expensive.

MoE instead uses an “on-demand” approach (i.e., conditional computation), letting only part of the experts participate and avoiding unnecessary resource waste.

1. Sparse activation and activated parameters

In theory, the FFN layer in a Transformer must fully compute for all tokens, but in practice only some neurons fire—showing natural sparsity (see MoEfication: Transformer Feed-forward Layers are Mixtures of Experts). MoE builds on this by splitting a large FNN into E “experts,” and having the router select only K experts per token:

2. How the router and routing strategies work

Expert selection is performed by the router (gating network). A typical implementation is:

1. Linear projection: H(x)=xW\_g
2. Noise injection: H'(x)=H(x)+g·StandardNormal()⊙SoftPlus(H(x)) (also called Noisy Gating)
3. Top‑K selection: G(x)=Softmax(KeepTopK(H'(x),K))

• Top‑2 vs. Top‑1: GShard uses Top‑2 and mixes outputs to improve robustness; Switch Transformer uses Top‑1 to simplify communication complexity (see GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding and Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity).
• Hierarchical sparsity: Models like Qwen3 dynamically adjust K at different depths to achieve more fine-grained conditional computation.

3. Load balancing and auxiliary loss

Some experts may be continuously activated due to higher scores, causing overload. To prevent uneven resource allocation, MoE introduces an auxiliary load loss (Auxiliary Load Loss) mechanism, usually defined as the coefficient of variation of the number of tokens each expert receives:

L\_aux = λ · CV(n₁, n₂, …, n\_E) In this formula, nᵢ is the number of tokens processed by the i-th expert in the current batch, and λ is a balancing weight.

This loss term is optimized together with the main loss during pretraining to keep expert utilization roughly balanced (see GShard and the transformers library’s aux\_loss implementations).

VI. Classic MoE model examples

Below are several representative open-source MoE models in chronological order:

1. Sparsely-Gated Mixture-of-Experts Layer (2017)

Proposed by Shazeer et al. (Google Brain) (see Outrageously Large Neural Networks: The Sparsely‑Gated Mixture‑of‑Experts Layer). It introduced the gating router mechanism into the feed-forward layer, used learnable noisy Top‑K gating (Noisy Top‑K Gating), and proposed auxiliary load loss to balance expert load. It established conditional computation as the foundation for sparse activation and efficient training at large scale.

2. GShard (2020)

Proposed by Lepikhin et al. (Google Research) (paper: GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding). It deployed Top‑2 gated MoE in every Transformer FNN layer, introduced the concept of expert capacity, automatically partitioned experts across multiple machines and GPUs, and used random routing and load balancing strategies. It enabled cross-machine expert parallelism and automatic sharding, supporting training and inference at 600B+ parameters and improving scalability.

3. Switch Transformer (2021)

Proposed by Fedus et al. (Google Research) (paper: Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity). It introduced Top‑1 gating: each token visits only one expert, reducing communication complexity. It also introduced the capacity factor (Capacity Factor) to balance expert load, and supported bfloat16 training. It achieved a good trade-off between communication overhead and throughput, scaling to 1T parameters with high efficiency.

4. GLaM (2021)

Proposed by Du et al. (Google Research) (paper: GLaM: Efficient Scaling of Language Models with Mixture-of-Experts). The model has 64 experts and activates 2 per layer. It uses a mixture-of-modality routing strategy, assigning tasks of different modalities (text, code, etc.) to different experts. Only 8% of parameters are activated. It shows that sparse activation can preserve performance and further improve multimodal and multitask learning.

5. DeepSeek-MoE (2024)

Proposed by the DeepSeek AI team (paper: DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models). It uses fine-grained expert segmentation to split the FFN into smaller experts, and introduces shared experts for common knowledge. Each token activates 6 experts. It provides new insights into expert specialization and redundancy reduction, showing the potential of multi-expert fusion and sharing.

6. LLaMA-MoE (2024)

Developed by the Sys4NLP team based on Meta LLaMA2 (see LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training). It supports multiple expert partitioning methods (random, clustering, co-activation graphs, etc.), offers both Top‑K noisy gating and Switch single-expert gating strategies, and trends toward lighter expert parameter designs. It demonstrates a path for MoE on lightweight models, offering feasible scaling strategies for mid- and small-parameter models.

7. Qwen3-235B-A22B (2025)

Released by Alibaba’s Tongyi Qwen team (see official docs and open-source repos). It has 235B parameters and 128 experts. It uses hierarchical sparse scheduling to dynamically adjust the number of activated experts; each token activates 8 experts. It supports a native 32K context length and can be extended to 128K with YaRN, and it uses GQA attention and asynchronous offloading to improve parallel compute efficiency. By combining hierarchical scheduling and dynamic activation, it provides a new solution for long-text processing and high-throughput scenarios, further reducing inference cost.

VII. Suitable scenarios and limitations of MoE

In real deployments, MoE and traditional Transformers each have strengths, and the choice should depend on available resources and needs:

• MoE is more suitable for multi-machine, high-throughput scenarios
  • Expert parallelism: Expert sub-networks can be split across multiple machines or devices, and each machine handles only a few experts—avoiding GPU memory pressure from copying the entire model.
  • Conditional routing: The router only needs to communicate necessary routing information rather than synchronizing all parameters, greatly reducing communication overhead and improving throughput.
  • Elastic scalability: As the compute cluster scales out, you can add or reassign experts. Model capacity is highly decoupled from hardware resources.
• Dense Transformers are more suitable for single-machine, low-memory, or low-throughput needs
  • Routing and scheduling overhead: In single-machine settings, the relative overhead of gating, expert state management, and cross-device communication can increase, potentially raising latency and implementation complexity.
  • Simpler deployment: Dense Transformers do not require extra load balancing or routing strategies; frameworks and code are mature, and it’s easier to trade off size vs. speed.
  • Resource efficiency: When GPU memory and compute are limited, pruning, quantization, or distillation can optimize dense networks for better cost performance.

VIII. Summary and outlook

Looking back at MoE’s development, we can see innovations from “noisy Top‑K gating” to “cross-machine automatic sharding,” and further to “single-expert routing,” “multimodal experts,” “fine-grained partitioning,” and “hierarchical sparse scheduling.” Each iteration pursues a balance between “larger model capacity” and “lower compute cost.” MoE’s core idea—conditional computation—brings great scalability and efficiency, but also introduces challenges in routing, load balancing, and communication.

MoE has become an important branch in large-scale deep learning. Understanding its design principles and application scenarios is crucial for advancing the next generation of efficient intelligent systems.

Some terms

Token: The smallest unit used by a model when processing text—for example, a Chinese character or an English word.

Parameters: Trainable numeric values in a neural network, like the number of LEGO bricks; more parameters generally mean a more complex model.

Conditional Computation: A mechanism that activates only part of the experts for computation based on the input—like consulting specialized teachers rather than holding an all-hands meeting.

Expert: A sub-network within an MoE model; each expert handles certain types of data or features, like teachers in different subjects.

Router/Gating Network: A module that decides which experts should process each input token, like a dispatcher assigning tasks.

Activated Parameters: The number of parameters that actually participate in computation during one forward pass—usually far fewer than the total.

Sparse Activation: Activating only a subset of experts or parameters per computation to save resources.

Dense Model: Traditional models that use all parameters each time—like everyone attending every meeting.

FLOPs (Floating Point Operations): A metric for computation; lower FLOPs means less compute cost.

GPU Memory: The GPU space used to store model parameters and intermediate results; lower usage makes deployment easier.

Feed-Forward Network (FFN): A neural network layer in Transformers responsible for nonlinear transformations.

Multi-Head Attention: A Transformer mechanism that allows the model to attend to different parts of the input simultaneously.

Top‑K Routing: A strategy where the router selects the K highest-scoring experts to participate in computation.

Noisy Gating: Adding noise during expert selection to improve robustness and load balancing.

Auxiliary Load Loss: A loss function used to balance expert workloads and prevent popular experts from becoming overloaded.

Coefficient of Variation (CV): A statistical measure of load balance across experts; smaller is more even.

Expert Parallelism: Distributing experts across multiple machines, each executing only its assigned sub-network.

Distributed Parallelism: Distributing model or data across multiple devices to compute in parallel for higher efficiency.

Pipeline Parallelism: Assigning different model parts to different devices, processing data step-by-step like an assembly line.

Tensor Parallelism: Parallelizing computation within a single layer across multiple devices.

Capacity Factor: A parameter controlling the maximum number of tokens each expert can process per forward pass to prevent overload.

Expert Capacity: The maximum number of tokens an expert can process in one forward pass.

Fine-Grained Expert Segmentation: Further splitting the FFN into more, smaller experts to increase specialization.

Shared Experts: Experts shared across tasks or inputs, mainly handling common knowledge.

Hierarchical Sparsity: Dynamically adjusting the number of activated experts at different network depths for fine-grained conditional computation.

GQA (Grouped Query Attention): An efficient attention mechanism that improves inference efficiency in large models.

Asynchronous Offloading: Moving part of the computation or data asynchronously to other devices to optimize resource utilization.

Mixture-of-Modality: Routing based on input modality (text, code, etc.) to different experts to improve multimodal processing.

Pruning: Removing less important parameters to reduce model size and compute.

Quantization: Using lower-precision data types for storing and computing parameters to reduce resource usage.

Distillation: Training a small model under guidance of a large model so it inherits capabilities.

Routing: The overall process of selecting experts for each token.

Main Loss: The primary optimization objective during training, such as cross-entropy.

Auxiliary Loss: A secondary objective used to improve specific traits (e.g., load balancing).

Expert Partitioning: Splitting a large network into experts, achievable via clustering or graph partitioning.

Co-activation Graph Partitioning: Partitioning experts based on activation correlations among neurons.

Gradient-aware Partitioning: Partitioning based on gradient information.

Expert Routing: Routing inputs to appropriate experts.

Top‑1/Top‑2 Routing: Assigning each token to the top 1 or top 2 experts.

Capacity: The maximum number of tokens each expert can process per forward pass.

Activation Ratio: The proportion of activated parameters in a forward pass.

Throughput: The amount of data a model can process per unit time; higher means more efficient.

Latency: The time a model takes to process an input; lower is better.

Automatic Sharding: Automatically distributing model parameters across devices for more efficient parallel compute.

bfloat16: A low-precision floating point format often used in large-model training to save memory.

YaRN: A technique to extend context length, improving long-text handling.