00 The two bills: training compute and inference memory

How many floating-point operations did it take to train DeepSeek-V3, how much memory does one 128K-token conversation take on GLM-4.5, and why did the second number start to drive model design in 2025-26?

Two bills

A model is paid for twice: once to train it, a bill counted in floating-point operations (FLOPs) and GPU-hours, and again for every token it serves. Training cost grows with the parameters each token touches and the number of training tokens. Serving cost at long context grows mostly with the bytes kept in memory and read back for every new token.

Training: from 6ND to context-driven cost

How much compute does training take? A good first answer counts how often each weight is used. For every token, each weight matrix takes part in three matrix products: one on the way forward and two on the way back.

Each product costs 2 FLOPs per weight: one multiply and one add. So every parameter costs about 6 FLOPs per training token, and a model with N_a active parameters trained on D tokens costs

The Chinchilla paper uses this formula as its compute budget. Inference runs only the forward pass, so serving a token costs about 2·N_a FLOPs, a third of training.

The count that matters is the parameters that actually multiply the token. In a dense model that is all of them. A mixture-of-experts (MoE) model splits each feed-forward block into E experts, and a router sends each token to only k of them (chapter 02). Unchosen experts do no arithmetic for that token, so FLOPs follow the active parameters N_a: the attention and output-head weights, any shared experts (experts every token uses; chapter 02) and the k chosen experts. The input embedding is a table lookup that costs no FLOPs, although some labs include it in their reported active counts. The total N matters for memory.

What 6ND leaves out. A transformer does two kinds of work for each token:

• Applying the learned weights. The token's vector is multiplied by weight matrices. This costs the same whether the context holds 10 tokens or 10,000, about 2N_a FLOPs per token in the forward pass, and it is all that 6ND counts.
• Attention. The token compares itself with the tokens before it: about 10 comparisons with 10 earlier tokens, about 10,000 with 10,000. This work grows with the context but adds no parameters, so a parameter count cannot see it.

For a standard dense transformer with L layers of width d, Kaplan et al. write the forward cost per token as two terms, with n the context length:

Their ratio says when attention matters:

When the context is short next to the width, attention is a small add-on. At d = 4,096 and 8K tokens it is about 17% of the weight work, which is why Kaplan et al. drop it when d > n/12. At 128K tokens the same model spends 2.7 times as much on attention as on its weights, and at 1M about 21 times. The parameter count did not change; the context did. So 6ND is a good budget for pretraining at short context and a poor one for long-context training and serving, the subject of chapter 03.

Inference: prefill and decode

A model writes one token at a time. In every layer, each new token attends to all the tokens before it: its query is compared with their keys, and their values are mixed according to the scores. Serving runs in two phases, which Pope et al. call prefill and decode:

• Prefill pushes all prompt tokens through the network together, so each weight matrix multiplies a large block of activations.
• Decode generates one token per step, since each step needs the previous one's output.

Both spend about 2·N_a FLOPs per token on weights. What separates them is how many bytes each step has to read, and most of those bytes come from the KV cache.

The KV cache

The earlier tokens do not change while the model writes, so neither do their keys and values. Recomputing them at every step would repeat the same work, so servers keep them in memory instead. That store is the KV cache. In each layer, a decode step does three things (Figure 0.1):

1. computes the new token's query, key and value;
2. scores the query against every cached key and mixes the cached values by the resulting weights;
3. appends the new key and value to the cache.

The cache trades computation for memory, and the memory has two costs:

• Capacity. The cache grows by one entry per token in every layer and must fit in GPU memory beside the weights, so it limits how many conversations a server can hold.
• Bandwidth. Step 2 reads the whole cache again for every generated token, so the bytes moved per step grow with the context.

Its size follows from what each layer stores: one key and one value for each key/value head.

A sequence of n tokens needs n times that, and a batch of B sequences B times more. With easy numbers, 32 layers, 8 KV heads of 128 numbers and 2-byte bfloat16 give 2 × 32 × 8 × 128 × 2 = 131,072 bytes (128 KiB) per token: 1 GiB at 8,192 tokens and 16 GiB at 131,072.

The formula counts KV heads h_kv, not query heads h. The original transformer gave every query head its own key/value head, and for a 500B+ model laid out that way, Pope et al. find that batch 512 at context 2,048 needs a 3 TB cache, three times the parameters. Multi-query attention shares one key/value head across all query heads, and grouped-query attention (GQA) uses an intermediate number of key/value heads, so each cached head serves h/h_kv query heads and the cache shrinks by that factor. Chapter 03 covers designs that shrink it further.

def decode_step(x, cache, layers):            # x: [d], the newest token's hidden state
    for l, layer in enumerate(layers):
        u = rmsnorm(x)
        q = (u @ layer.W_q).reshape(h, d_h)     # [h, d_h]   (RoPE omitted)
        k = (u @ layer.W_k).reshape(h_kv, d_h)  # [h_kv, d_h]
        v = (u @ layer.W_v).reshape(h_kv, d_h)  # [h_kv, d_h]
        cache.K[l].append(k)                    # cache grows by 2*h_kv*d_h numbers
        cache.V[l].append(v)
        K = stack(cache.K[l])                   # [n, h_kv, d_h]: all n tokens are read
        V = stack(cache.V[l])                   # [n, h_kv, d_h]
        g = h // h_kv                           # query heads per KV head
        heads = []
        for i in range(h):
            j = i // g                          # the KV head this query head uses
            scores = K[:, j, :] @ q[i] / sqrt(d_h)   # [n], one per key position s
            heads.append(softmax(scores) @ V[:, j, :])  # [d_h]
        x = x + concat(heads) @ layer.W_o       # [d]
        x = x + layer.ffn(rmsnorm(x))           # dense FFN or MoE
    return x

Why decode is limited by memory bandwidth

Caching saves recomputation, but every generated token must read the whole accumulated cache. Whether that matters depends on how fast a chip reads memory compared with how fast it does arithmetic. Dividing one by the other gives the FLOPs each byte read from high-bandwidth memory (HBM) must feed to keep the chip busy. A TPU v4 chip does 275 TFLOPS of bfloat16 matrix arithmetic and reads 1,200 GB/s, about 229 FLOPs per byte. An H100 SXM does 989 TFLOPS of dense 16-bit matrix arithmetic and reads 3.35 TB/s, about 295.

• Weights are shared. A 2-byte weight read once and applied to B tokens does 2B FLOPs, B per byte. Prefill applies each weight to thousands of prompt tokens, so it keeps the chip busy and is compute-bound. Decode has one token per sequence, but batching B sequences lets one read serve B tokens.
• The cache is not. Each sequence's keys and values serve only that sequence, so batching does not help. With GQA each cached number feeds h/h_kv query heads at 2 FLOPs each, far below the 229 to 295 a chip needs per byte.

So decode is usually memory-bandwidth-bound: the bytes read per step set its speed, and at long context most of those bytes are cache (Figure 0.2). That is the common case rather than a law. A large batch of short sequences can push decode back toward compute, and some attention designs raise the FLOPs per cached byte (chapter 03).

Why memory became the binding constraint

The training bill is paid once, however long users' conversations later get; the serving bill grows with every token of context. An agent keeps its task, opened files, tool outputs and reasoning in context across many steps, and Kimi K3's report describes training for million-token agentic RL (reinforcement learning on multi-step tool-using tasks, where each rollout keeps the whole session in context). Each session owns a cache that grows linearly with n. Capacity caps how many sessions fit on a server, and so the throughput; bandwidth sets the time per token. The 2025-26 reports name this cause and report cache sizes at 1M tokens.

Many of the later chapters shrink the term 2·L·h_kv·d_h·b·n: compress each layer's entry (multi-head latent attention, MLA), let most layers see only a recent window, replace the growing cache with a fixed-size state (linear attention), read only selected entries (sparse attention), or store fewer bytes per number. DeepSeek-V3's MLA caches 512 + 64 = 576 numbers per layer per token (a 512-number compressed latent plus a 64-number position key; chapter 03) where GLM-4.5 caches 2 × 8 × 128 = 2,048, so its 128K cache is 61 × 576 × 2 × 131,072 ≈ 9.2 GB instead of 49.4 GB. Hybrids shrink L, the number of layers that keep a cache. Qwen3.5 (397B total) has 60 layers, but only its 15 gated-attention layers cache keys and values, with 2 KV heads of dimension 256; the other 45 are Gated DeltaNet linear-attention layers that keep a fixed state (chapter 03). Its 128K cache is 15 × 2 × 2 × 256 × 2 × 131,072 ≈ 4.0 GB, plus about 94 MB of state (45 layers × 64 heads × 128 × 128 numbers at 2 bytes) that does not grow with n.

Who uses it and why

"Concurrently, the emergence of long-horizon scenarios and tasks — from complex agentic workflows to massive cross-document analysis — has also made efficient support for ultra-long contexts critical for future progress."

"The widespread adoption of long-horizon agents has made model workloads increasingly input-heavy. Although prior work has substantially reduced the cost of long-context computation, prefill remains computationally expensive, and large KV caches continue to strain HBM and SSD capacity and data-transfer bandwidth."

"Kimi Linear outperforms full MLA with a sizeable margin across all evaluated tasks, while reducing KV cache usage by up to 75% and achieving up to 6× decoding throughput for a 1M context."

1. Kaplan et al., Scaling Laws for Neural Language Models (2020), §2.1 and Table 1
2. Hoffmann et al., Training Compute-Optimal Large Language Models (Chinchilla, 2022), §3.3
3. Pope et al., Efficiently Scaling Transformer Inference (2022), §2 and §4
4. Shazeer, Fast Transformer Decoding: One Write-Head is All You Need (2019)
5. Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints (2023)
6. DeepSeek-AI, DeepSeek-V3 Technical Report (2024), §1 Table 1, §4 and DeepSeek-V3 config.json
7. GLM-4.5 Team, GLM-4.5 report (2025), §1-2 and GLM-4.5 config.json
8. Meta, Llama 4 model card and Llama 4 blog
9. NVIDIA H100 product page, technical specifications
10. DeepSeek-AI, DeepSeek-V4 report (2026), §1
11. DeepSeek-AI, DeepSeek-V4.1-Flash report (2026), Abstract
12. Kimi Team, Kimi K3 report (2026), §1
13. Kimi Team, Kimi Linear report (2025), Abstract
14. gpt-oss-120b config.json

01 Scale and sparsity

How large are the 2025-26 base models, how much of each one does a single token use, and why were the large MoE bases trained on 20 to 140 times more tokens per active parameter than Chinchilla's compute-optimal ratio for dense models (and small dense models even more)?

Total and active parameters

In a dense transformer every parameter processes every token, so the total count N equals the active count N_a. Gemma 3 27B is dense: of its 25,600M non-embedding, 1,416M embedding and 417M vision-encoder parameters, all the non-embedding ones serve each text token (the embedding table is a lookup and the vision encoder runs only on images).

A mixture-of-experts (MoE) model pulls the two counts apart: each feed-forward block becomes E experts, and a router picks k of them per token (chapter 02), what Shazeer et al. called conditional computation, where parts of the network are active on a per-example basis. More experts raise N; N_a depends only on k and the expert size.

A toy with easy numbers shows how far apart they can drift: 4B of shared weights (attention and the other non-expert layers) that every token uses, and 96B of experts, 64 per layer, with k = 4 per layer.

Doubling the experts to 128 per layer, each the same size, makes the total 4B + 192B = 196B, while a token still uses 4B + 192B × 4/128 = 10B.

The ratio of the two counts is the active share:

It is 100% for a dense model and falls from 10% to 10/196 ≈ 5.1% as the toy's experts double. The two counts pay for different things (chapter 00):

• Active parameters pay for compute. The training bill is 6·N_a·D for D training tokens, and serving costs 2·N_a FLOPs per token.
• Total parameters pay for memory and capacity. Any token may be routed to any expert, so all N must stay loaded: the toy's doubling takes its bfloat16 weights from 200 GB to 392 GB at the same FLOPs. In return, Kimi's K2 report finds that at a fixed active count, more experts lower the loss (quoted below).

Capacity is bought with memory rather than compute, which is why totals can grow much faster than active counts.

def moe_ffn(x, router, experts, k):          # x: [d], one token's hidden state
    scores = router(x)                         # [E], one score per expert
    top = argsort(scores)[-k:]                 # the k highest-scoring experts
    w = softmax(scores[top])                   # [k] mixing weights (one common choice)
    y = zeros_like(x)
    for wi, e in zip(w, top):                  # only k of the E experts run, so the
        y += wi * experts[e](x)                # token pays FLOPs for k experts, not E
    return y                                   # but all E experts must stay in memory

How sparse the 2025-26 bases are

Real bases sit near or below the doubled toy's 5.1%. DeepSeek-V3 runs 37B of 671B per token (5.5%), gpt-oss-120b 5.13B of 116.83B (4.4%), Kimi K3 104.2B of 2.78T (3.7%) and Kimi K2 32.6B of 1.04T (3.1%). Kimi K2 in bfloat16 needs 1.04T × 2 bytes ≈ 2.1 TB for its weights while doing the arithmetic of a 32.6B dense model.

In Figure 1.1, read each outline as memory and its filled sliver as compute per token: the totals run from 27B to 2.78T, while the MoE active counts stay between 5B and 41B, Kimi K3's 104.2B aside.

Beyond these nine, every new base above 32B parameters in the grid these notes were first written from uses experts. Of its 31 new pretraining runs, 21 are MoE, 3 mix dense and MoE layers, and the 7 dense ones have at most 32B parameters. Among the labs added later, the one exception is NVIDIA's Nemotron-H-56B (Mar 2025), which has no experts: about 8% of its layers are attention, and the rest alternate Mamba-2 state-space layers (chapter 03) with dense feed-forward layers.

Compute-optimal: about 20 tokens per parameter

How long should a model train? For dense models the question is how to split a fixed training budget C ≈ 6ND between parameters and tokens.

The Chinchilla paper (Hoffmann et al., 2022) trained over 400 models from 70M to over 16B parameters and concluded that for every doubling of model size the number of training tokens should also be doubled: its three methods give N and D each growing as about C^0.5, so doubling the budget scales both by √2 ≈ 1.4 and keeps their ratio. Kaplan et al. had found the optimal model size growing as C^0.73 and the tokens as C^0.27, which put most extra compute into larger models. The paper's Table 3 puts a 1B model at 20.2B tokens and a 10B model at 205.1B, about 20 tokens per parameter, and Chinchilla itself, 70B parameters on 1.4T tokens, sits at exactly 20.

Compute-optimal answers one question: the lowest training loss for a given training budget. It counts nothing that happens after training, and it was fitted on dense models. The next section adds both.

Over-training on purpose

Sardana et al. add the serving bill. A model trained on D_train tokens and then used on D_serve tokens costs

If D_serve = 10D_train, for example, serving costs 20 · N · D_train, over three times the training bill. When D_serve is large, a smaller N trained on more tokens reaches the same loss for less total compute, even though its training run alone is less efficient. Sardana et al. conclude that labs expecting large demand should train models smaller and longer than Chinchilla-optimal, and their models kept improving up to 10,000 tokens per parameter. LLaMA made the same case in 2023: a smaller model trained longer will ultimately be cheaper at inference.

Experts strengthen the argument. In an MoE model both terms scale with N_a, so a lab can keep N_a small for cheap tokens, grow the total for capacity, and train on as many tokens as its data supports. The ratio to watch is therefore D/N_a, tokens per active parameter. Chinchilla's 20 was fitted on dense models and no comparable value is established for MoE models, so the multiples of 20 below are a rough guide to how far labs go past the dense rule, not an exact measure of over-training.

def lifetime_flops(N_a, D_train, D_serve):     # Sardana et al., with N replaced by N_a
    return 6 * N_a * D_train + 2 * N_a * D_serve

Figure 1.2 widens this to twelve releases on a log scale: the MoE bases sit 20 to 140 times past Chinchilla's dashed 20, and the small dense Gemma 3 270M far beyond, at 22,222.

New bases and refreshes

Releases come in two kinds:

• A new pretraining run trains a base from scratch and pays the whole training bill. DeepSeek-V3's pre-training took 2,664K of its 2,788K GPU-hours; post-training took 5K.
• A refresh starts from an existing base, changing only post-training or continuing its pretraining.

Labs therefore train bases rarely and release on top of them often. Of the 68 releases in that grid, 31 are new pretraining runs; 19 change only post-training, 10 continue pretraining an existing base, 2 are speculative-decoding drafters, 1 is a distilled student and 5 do not say. A drafter is a small model that proposes tokens for the large one to check (chapter 06); a distilled student is trained to imitate another model's outputs (chapter 12).

DeepSeek-V3's base, released in December 2024, carried nine more releases up to V3.2 in December 2025. Two of them added continued pretraining: 840B tokens of long-context training for V3.1, 5.7% of the base's 14.8T, and a phase that added sparse attention for V3.2-Exp. A refresh can change data, context length and even attention, but it keeps the base's N and N_a.

Who uses it and why

"In MoE models, a single token activates only a fraction of the total parameters. MoE architectures are more compute efficient for training and inference and, given a fixed training FLOPs budget, delivers higher quality compared to a dense model."

"we observe that — under a fixed number of activated parameters (i.e., constant FLOPs) — increasing the total number of experts (i.e., increasing sparsity) consistently lowers both the training and validation loss, thereby enhancing overall model performance"

"Meanwhile, its small activation footprint yields low inference latency and serving cost."

1. Shazeer et al., Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017)
2. Kaplan et al., Scaling Laws for Neural Language Models (2020)
3. Hoffmann et al., Training Compute-Optimal Large Language Models (Chinchilla, 2022), Abstract, Tables 2-3
4. Sardana et al., Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws (2024)
5. Touvron et al., LLaMA: Open and Efficient Foundation Language Models (2023), §1
6. DeepSeek-AI, DeepSeek-V3 Technical Report (2024)
7. Kimi Team, Kimi K2 report (2025) and Kimi K3 report (2026)
8. OpenAI, gpt-oss-120b and gpt-oss-20b model card (2025)
9. Gemma Team, Gemma 3 Technical Report (2025)
10. Meta, Llama 4 blog and Llama 4 model card
11. DeepSeek-AI, DeepSeek-V4.1-Flash report (2026) and DeepSeek-V4 report (2026)
12. DeepSeek-AI, DeepSeek-V3.2-Exp report (2025) and DeepSeek-V3.1 release note
13. Other token and parameter counts in Figures 1.1-1.2: GLM-4.5, GLM-5, MiMo-V2-Flash, Inkling, MAI-Base-2, Gemma 3 model card (270M)
14. Atlas, pretraining section (coded grid of 68 releases)
15. NVIDIA, Nemotron-H report (2025)

02 Inside a mixture-of-experts layer

How can DeepSeek-V3 store 671B parameters but run only 37B of them for each token, and what keeps its 256 experts evenly used?

From one feed-forward block to many experts

A transformer block has two parts: attention, where tokens read each other (chapter 03), and a feed-forward network (FFN), which transforms each token on its own. A standard block sends every token through the same FFN, so every FFN weight works on every token. DeepSeek-V3's FFN is a SwiGLU block with three matrices and 3 · d · d_ff weights, for model width d and inner width d_ff.

A mixture-of-experts (MoE) layer replaces that FFN with E smaller FFNs, called experts, and a router that sends each token to k of them:

• Memory follows E. Every expert is stored, used or not.
• Compute follows k. A token is multiplied only by the experts it visits.

That is how DeepSeek-V3 holds 671B parameters in total (N) but runs 37B per token (N_a).

One token through the layer

The router holds one learned vector e_i ∈ ℝ^d per expert and takes four steps for each token u:

1. Score the token against every expert: s_i = σ(u^⊤e_i), i = 1…E. DeepSeek-V3 uses this sigmoid; Switch Transformer used a softmax.
2. Choose the k best: Top(u) = TopK({s_i + b_i}, k). Switch used k = 1 and GShard top-2 gating. The bias b_i is DeepSeek-V3's balancing term, explained below.
3. Weight each chosen expert by its share of the chosen scores: g_i = s_i / ∑_j∈Top(u) s_j.
4. Combine the gated expert outputs with the N_s shared experts, which every token uses without routing, and the residual input u:

with u, h′ ∈ ℝ^d, s, b ∈ ℝ^E and |Top(u)| = k. The gate is what trains the router: each chosen expert's output is scaled by its score, so the loss gradient can raise or lower that score. Figure 2.1 draws the same steps for one token.

A toy layer. Take E = 4 routed experts, k = 2 and four tokens, with no bias yet:

A's gates are 0.9/1.5 and 0.6/1.5. Each token runs two of the four stored experts. Now count the load: expert 1 gets all four tokens and expert 4 none, where a fair share is 4 × 2/4 = 2 each.

More, smaller experts, plus a shared one

Early MoE layers used a few large experts, often 8 or 16. The DeepSeekMoE paper (2024) argued that each such expert must store mixed knowledge, while knowledge that every token needs gets copied into many experts. It proposed two fixes:

• Fine-grained segmentation splits each expert into several narrower ones (the paper's example uses 4, each a quarter as wide) and activates proportionally more of them, so parameters and compute stay fixed while the number of expert combinations explodes: top-2 of 16 experts allows 120, top-8 of 64 allows 4,426,165,368.
• Shared expert isolation adds experts that every token uses without routing, to hold the common knowledge.

DeepSeek-V3 has 256 routed experts of width 2048, 8 active per token, and 1 shared expert.

Keeping the load even

Nothing in the language-modeling loss spreads tokens out, and imbalance like the toy's costs twice:

• Quality. Experts that win early get more gradient and win more, while the rest starve: routing collapse.
• Speed. The GPU holding a popular expert makes the others wait.

An auxiliary loss is the classic fix. Switch Transformer adds

which is smallest under uniform routing, with α = 10⁻². Wang et al. (2024) showed the bind: a small α lets routing collapse, and a large α adds gradients that conflict with language modeling and hurt perplexity.

A routing bias, their auxiliary-loss-free method, adds a per-expert bias b_i to the scores only when choosing the top k. Gates use the raw scores, so the bias changes which experts run but leaves the output weights and gradients alone. After each step an overloaded expert's bias falls by γ and an underloaded one's rises by γ. The update uses only the finished batch, so no routing decision depends on future tokens.

In the toy, one update with γ = 0.2 gives b = (−0.2, 0, 0, +0.2). C's biased scores become (0.2, 0.6, 0.1, 0.5) and D's (0.1, 0.1, 0.7, 0.4), so both swap expert 1 for expert 4 and every expert gets 2 tokens. C's gates still come from its raw scores, 0.6 and 0.3: 2/3 and 1/3.

At 1B parameters this improved validation perplexity from 9.56 to 9.50 and cut MaxVio, the busiest expert's excess over a balanced load, from 0.72 to 0.04.

def moe_layer(u, W_r, b, experts, shared, k):
    # u: [B_tok, d] tokens; W_r: [E, d] expert vectors; b: [E] routing bias
    s = sigmoid(u @ W_r.T)                   # [B_tok, E] affinity scores
    top = topk_indices(s + b, k)             # [B_tok, k] bias only picks experts
    g = take_along_axis(s, top, axis=-1)     # [B_tok, k] gates use raw scores
    g = g / g.sum(axis=-1, keepdims=True)    # normalize over the k chosen
    y = shared(u)                            # [B_tok, d] always-on shared expert
    for t in range(len(u)):
        for j in range(k):
            y[t] += g[t, j] * experts[top[t, j]](u[t])
    load = bincount(top.ravel(), minlength=len(b))   # [E] tokens per expert
    return u + y, load

def update_bias(b, load, gamma=1e-3):
    # after the optimizer step; the new b routes the next batch
    return b + gamma * sign(load.mean() - load)      # lift idle, damp busy

DeepSeek-V3 uses γ = 0.001 for the first 14.3T tokens and 0 for the last 500B, plus a sequence-wise loss with α = 0.0001 against extreme imbalance inside one sequence. Balancing at batch level leaves each sequence free to favor the experts for its domain. In DeepSeek's 1B tests validation loss was 2.258 with a sequence-wise loss and 2.253 with the bias or a batch-wise loss. Microsoft AI found the same for MAI-Thinking-1: an f·P auxiliary loss in GShard's form, with f and P counted over the whole global batch (all data-parallel workers and micro-batches) instead of per sequence, matched the loss-free method.

Quantile Balancing. Kimi K3 (2026) raised the count to 896 routed experts and found the fixed step too blunt: a small γ adapts slowly and a large one makes loads oscillate. Quantile Balancing sets each bias in one shot, to the value that would have given that expert exactly its fair share of the batch just routed. In K3's toy example, 8 tokens, 4 experts and k = 1 start with loads (4, 3, 1, 0) and end with 2 each. K3 reads the quantile from per-GPU histograms summed in one all-reduce, applies the bias from the next step, and freezes it at inference.

How sparse should a layer be?

Kimi K2 defines sparsity as E/k; the toy's is 2. More experts at fixed k cost memory but no extra expert compute per token, and the labs found they pay off:

• Kimi K2. In small controlled runs with k = 8 and one shared expert, sparsity 48 needed 1.69×, 1.39× and 1.15× fewer FLOPs than sparsity 8, 16 and 32 to reach validation loss 1.5.
• Microsoft AI saw steady gains from 256 to 1,024 experts at top-8 and chose 512.
• Qwen went from 128 experts in Qwen3 to 512 in Qwen3-Next and its successors, reporting that with a global load-balancing loss, more expert parameters at a fixed active count steadily lower training loss.
• Ant set Ling 2.0's 8 of 256 experts with scaling laws that predicted over 7× the efficiency of a dense model, then activated 8 of 512 in Ling 3.0.

A latent MoE makes each expert cheaper by running the routed experts in a narrower space. NVIDIA's LatentMoE in Nemotron 3 Super (Mar 2026) runs them in a 1,024-wide latent space instead of the 4,096-wide hidden state and spends the savings on 512 experts with 22 active. Kimi K3 did the same four months later: it projects each token from width 7,168 down to a 3,584-wide latent vector once, sends that vector to its routed experts, and projects their combined output back up; shared experts keep the full width. Every copy sent to an expert, and every expert's input and output matrix, is half as large, which lets K3 afford 16 of 896.

When dense wins

An MoE buys loss per FLOP with memory. Meta's Muse Glimmer (2026) had to fit a 32GB memory budget, so Meta compared ladders (families of models of increasing size built by one rule) for a dense design and two MoE designs with 1 of every 8 or 16 experts active, holding inference memory fixed. At equal memory an MoE token uses only part of the parameters and so costs less compute; within the same GPU-hours the MoE ladders trained on 2,800 and 6,000 tokens per parameter against 550 for dense, and dense was still 2–2.5× more compute-efficient. Meta chose a dense 30B model.

Who uses it and why

"To achieve a better trade-off between load balance and model performance, we pioneer an auxiliary-loss-free load balancing strategy (Wang et al., 2024a) to ensure load balance."

"Though increasing sparsity leads to better performance, this gain comes with increased infrastructure complexity. To balance model performance with cost, we adopt a sparsity of 48 for Kimi K2, activating 8 out of 384 experts per forward pass."

"Scaling curves (Fig 1) demonstrated that within constraints of training compute and parameter size, dense configurations yielded a 2×-2.5× increase in compute efficiency compared to the MoE counterparts."

1. Fedus, Zoph, Shazeer. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity (2021)
2. Lepikhin et al. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020)
3. Dai et al. DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models (2024)
4. Wang et al. Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts (2024)
5. DeepSeek-AI. DeepSeek-V3 Technical Report (2024) and config.json
6. DeepSeek-AI. DeepSeek-V4 Technical Report (2026)
7. Kimi Team. Kimi K2: Open Agentic Intelligence (2025)
8. Kimi Team. Kimi K3: Open Frontier Intelligence (2026)
9. Microsoft AI. MAI-Thinking-1: Building a Hill-Climbing Machine (2026) and Frontier Pre-training at Microsoft AI
10. Meta. Muse Glimmer 30B: An On-Device Agentic Model (2026)
11. NVIDIA. Nemotron 3 Super report (2026) and Nemotron 3 Nano report (2025). Ant. Ling 2.0 report (2025) and Ling-3.0-flash-base card. Qwen. Qwen3-Next blog and Qwen3.8-Flash-Next blog. StepFun. Step 3.5 Flash report (2026).

03 Attention and the KV cache

Why does every token a model has already read cost memory while it generates, and what did the labs change in attention to make 128K to 1M-token contexts affordable?

What a new token keeps and reads

In multi-head attention (MHA), each of h heads has its own queries, keys and values of size d_h:

Every head caches its own keys and values (h_kv = h), so chapter 00's formula becomes

Chapter 00's toy model (32 layers, heads of 128 numbers, bfloat16) with 32 such heads needs 2 × 32 × 32 × 128 × 2 = 524,288 bytes (512 KiB) per token, 64 GiB at 131,072 tokens. That cache costs capacity, since it must sit in GPU memory, and bandwidth, since every new token reads all of it. Each design below changes one thing:

• What a layer stores per token: GQA and MQA share key/value heads; MLA stores one compressed vector.
• How many tokens a layer keeps: sliding windows, compressed attention and linear attention.
• How many entries a new token reads: sparse attention, which saves bandwidth and arithmetic but not capacity.

Share keys and values: MQA and GQA

Multi-query attention (MQA) keeps h query heads but one key/value head shared by all of them, shrinking the cache h-fold with "only minor quality degradation" in Shazeer's tests. Grouped-query attention (GQA) splits the query heads into h_kv groups that each share one key/value head; GQA-1 is MQA and GQA-h is MHA. A new token still reads every cached token, but each entry is h/h_kv times smaller: in the toy, 16 GiB at 128K with 8 KV heads and 2 GiB with MQA.

GQA's authors converted MHA checkpoints by mean-pooling each group's heads and training for 5% of the original compute. GLM-4.5 uses 96 query and 8 KV heads; gpt-oss uses 64 and 8. StepFun's Step-3 (Jul 2025, 321B) went to the MQA end: its 64 query heads share one key and one value head of dimension 256, and its queries are factorized through a low-rank bottleneck (MFA). Step 3.5 Flash moved to GQA with 8 KV heads.

Figure 3.1 compares what MHA, GQA and MLA keep per token in one layer.

Compress keys and values: MLA

Multi-head latent attention (MLA), from DeepSeek-V2, keeps h distinct key and value heads but stores none of them. It compresses each hidden state x_t into a short latent vector, from which fixed matrices rebuild every head's keys and values:

The cache holds the latent (d_c numbers) plus a small position key k^R (d_r numbers) shared by all heads. A new token still reads every cached token, but each entry shrinks from 2hd_h numbers per layer to d_c + d_r. With DeepSeek-V3's d_c = 512 and d_r = 64, the toy model would store 576 numbers per layer, 36 KiB per token and 4.5 GiB at 128K, a fourteenth of MHA.

During decoding the keys and values are never rebuilt: the expansion matrices fold into the query and output projections, so all heads read the same cached latents, which DeepSeek calls the MQA mode of MLA. Rotary position embedding (RoPE) would block that folding, hence the separate position key.

Absorbed MLA trades memory for arithmetic: every head scores every cached latent over d_c + d_r dimensions and mixes it over d_c, 1,088 multiply-adds per cached token and head in DeepSeek-V3's shape, against 256 for a 128-number MHA head. With 128 heads sharing each 1,152-byte bfloat16 entry, that is about 240 FLOPs per cached byte, against 12 for GLM-4.5 (chapter 00) and near the ~295 an H100 needs, so the head count drives long-context cost.

Kimi K2 kept MLA but used 64 heads instead of DeepSeek-V3's 128: at 128K tokens, doubling heads adds 83% inference FLOPs for only 0.5–1.2% lower validation loss. Mistral Large 3 (Dec 2025) and Small 4 also use MLA-type attention, with latents of 512 and 256 numbers per layer in their configs.

Look only nearby: sliding windows and sinks

A sliding-window layer keeps fewer tokens instead of smaller entries: it attends only to the last w tokens, so its cache stops growing at w and each new token reads at most w entries. Older tokens are out of its reach, so models interleave window layers with global (full) layers, in a ratio written local : global. In the toy with 8 KV heads, a 3 : 1 mix with w = 1,024 keeps 4 GiB in its 8 global layers and 96 MiB in its 24 window layers, about 4.1 GiB instead of 16.

• Gemma 3 uses 5 : 1 (5 local layers per global one) with w = 1024. For a 2B model at 32K tokens, its report shows cache overhead falling from 60% of memory (global only) to under 15% with three local layers per global one and w = 1024.
• gpt-oss alternates 1 : 1 with w = 128.
• MiMo-V2-Flash uses 5 : 1 with w = 128: 39 window and 9 global layers, because its first layer is global and the first block has only four window layers (config hybrid_layer_pattern).
• StepFun's Step 3.5 Flash uses 3 : 1 with w = 512, choosing windows over linear attention to maximize decoding efficiency, and puts a sigmoid gate on each head's output.

Small windows expose a quirk of softmax: weights must sum to one, so a head with nothing to read still puts attention somewhere. StreamingLLM found models park it on the first tokens, which every later token sees, and called them attention sinks. Window attention collapses once they leave the window; keeping 4 initial tokens fixes it. gpt-oss instead gives each head a learned bias in the softmax denominator, "which enables the attention mechanism to pay no attention to any tokens". MiMo-V2-Flash copies this on its window layers (config add_swa_attention_sink_bias):

In MiMo's 32B ablation a w = 128 hybrid scored 54.9 on MMLU without sinks, 58.3 with them, and all-global attention 57.3.

Figure 3.2 compares which keys each query reads under each scheme.

Read only what matters: sparse attention

A window chooses what to read by position; sparse attention chooses by content. It keeps the whole cache but reads only a few entries per new token. DeepSeek Sparse Attention (DSA), introduced in DeepSeek-V3.2-Exp and kept in V3.2, adds a lightning indexer that scores each earlier token s for the current token t with H_I small heads:

Main attention then reads only the m = 2048 highest-scoring entries, so its work per token falls from O(n) to O(m). The indexer still scans all n, but it "has a small number of heads and can be implemented in FP8". DSA uses MLA's MQA mode, so each selected latent serves every head. At 128K that is one cached latent in 64.

def dsa_decode_step(x_t, latents, idx_keys, m=2048):
    # latents:  [t, d_c + d_r]  MLA cache, shared by all heads
    # idx_keys: [t, d_I]        one small indexer key per past token
    q_I, w_I = indexer_queries(x_t)          # [H_I, d_I], [H_I]
    scores = relu(q_I @ idx_keys.T)          # [H_I, t]  few heads, FP8
    I = (w_I[:, None] * scores).sum(axis=0)  # [t] one score per past token
    sel = top_k(I, min(m, len(I)))           # indices of m tokens
    return mla_attention(x_t, latents[sel])  # core attention reads m entries

DSA was added to dense-attention checkpoints by continued training, in two stages from DeepSeek-V3.1-Terminus:

1. Train only the indexer, on 2.1B tokens: its scores, softmaxed over past tokens, were pulled by a KL loss toward the main attention weights summed over heads and renormalized, so it learns which tokens real attention reads.
2. Switch on selection and train everything on 943.7B tokens at 128K.

GLM-5 added DSA the same way and reports 1.5–2× less attention compute on long sequences. GLM-5.2 trims the indexer: IndexShare picks top-k once per 4 layers, for 2.9× fewer per-token FLOPs at 1M.

Three more labs select blocks of tokens instead of single tokens, on GQA instead of MLA:

• MiniMax Sparse Attention in M3 (Jun 2026) picks 16 blocks of 128 tokens per KV group and keeps full attention only in the first 3 of 60 layers; M3.1-Flash makes all 60 sparse.
• Qwen Sparse Attention in Qwen3.8-Flash-Next fills a 2,048-token budget from 4-token blocks and was added by continued pretraining that, as at DeepSeek, first trained only the indexer.
• StepFun's Sparse GQA in Step 5 Preview picks 4,096 tokens in 8-token blocks, which cuts the cost of indexing and top-k selection 8×.

Store fewer entries: compressed attention

DeepSeek-V4 merges consecutive tokens into single cache entries, with two layer types:

• CSA (Compressed Sparse Attention) layers merge every 4 tokens' KV entries into one entry, a weighted sum whose softmax weights are computed from the tokens, and DSA then picks the top 1024 entries (V4-Pro).
• HCA (Heavily Compressed Attention) layers merge every 128 tokens into one entry and read all of them.
• After the first two layers (HCA in V4-Pro, pure sliding window in V4-Flash), the two types alternate by layer. Every layer also attends to the last 128 uncompressed tokens and has sinks.

At 128K tokens a V4-Pro CSA layer keeps 32,768 entries and reads 1,024, an HCA layer keeps and reads 1,024, and both read the last 128 tokens exactly. Together with FP8 storage for most KV dimensions (which alone nearly halves the cache), they bring the cache to about 2% of a BF16 GQA baseline (8 KV heads of dimension 128) at 1M tokens.

Keep a fixed-size memory: linear attention

Linear attention removes the growing cache. Each head keeps one fixed-size matrix, its state S, which every new token updates with its key and value and reads with its query, so memory and work per token are fixed, whatever n is. In the toy, a 128 × 128 state per head is 32 KiB in bfloat16: 1 MiB for a 32-head layer at any length, against 512 MiB at 128K for a GQA layer with 8 KV heads.

The state comes from regrouping attention. Katharopoulos et al. replaced the softmax similarity exp(q·k) with a plain product φ(q)·φ(k). Take φ as the identity and drop the normalizer, as is now common. Because matrix products are associative, the output can be regrouped:

Attention becomes a recurrent network whose state S replaces the cache.

The catch is recall. Reading with a stored key k_s returns S_tk_s = ∑_s′ v_s′(k_s′·k_s), which equals v_s only if k_s has unit length and every other key is orthogonal to it. d_k-dimensional keys allow that for at most d_k tokens, and S can add associations but never remove them, so stored values blur. The DeltaNet paper calls these key "collisions".

Two changes to the write sharpen recall:

• The delta rule (DeltaNet) first reads what memory returns for k_t and moves it toward v_t: S_t = S_t−1 + β_t(v_t − S_t−1k_t)k_t^⊤, one gradient step on ½‖Sk_t − v_t‖².
• Gating first multiplies S by a learned decay α_t ∈ (0, 1).

Gated DeltaNet uses both, since "gating enables rapid memory erasure while the delta rule facilitates targeted updates". Kimi Delta Attention (KDA), from Kimi Linear, gives each key channel its own decay. Figure 3.3 shows one step.

def gated_delta_step(S, q, k, v, alpha, beta):
    # S: [d_v, d_k] state; q, k: [d_k] (k L2-normalized); v: [d_v]
    # alpha: scalar (Gated DeltaNet) or [d_k] vector (KDA); beta in (0, 1)
    S = S * alpha                         # 1 forget (per column for KDA)
    v_old = S @ k                         # what memory returns for key k
    S = S + beta * outer(v - v_old, k)    # 2+3 erase old value, write new
    o = S @ q                             # read with the query
    return S, o

A fixed state cannot hold every detail of a long document, so production models keep some full-attention layers, in a ratio written linear : full. A 3 : 1 hybrid of the toy keeps 4 GiB at 128K in its 8 full layers plus 24 MiB of state in the other 24.

Kimi Linear repeats 3 KDA layers and 1 MLA layer (3 : 1); in its ablation 3 : 1 had the lowest validation perplexity, 5.65, ahead of 1 : 1 (5.66, but slower), 7 : 1 (5.70), all-MLA (5.77) and 15 : 1 (5.82). With the same 1.4T-token recipe as a full-MLA baseline, it scored 51.0 vs 47.2 on MMLU-Pro and 84.3 vs 81.3 on RULER at 128K, with up to 75% less KV cache and up to 6× decoding throughput at 1M (per-token time 6.3× faster in its Figure 1).

Other models mix differently:

• Kimi K3 scales Kimi Linear's layout to 69 KDA and 24 gated MLA layers (MLA whose output is multiplied by a learned, token-dependent sigmoid gate): 23 blocks of 3 : 1 plus one extra MLA layer that makes the last layer global.
• GLM-5.3-Flash pairs 3 KDA layers with 1 DSA layer: 34 and 11 in its config, 11 blocks plus one final KDA layer, so even its full layers are sparse.
• MiniMax-Text-01 (Jan 2025, 456B) already put one softmax layer after every seven lightning-attention (linear) layers.
• Qwen3-Next (Sep 2025) mixed Gated DeltaNet and gated attention at 3 : 1, and Qwen kept that layout up to Qwen3.8's 2.4T parameters.
• Ant turned the GQA layers of its finished Ling 2.0 base into 7 : 1 Lightning and MLA layers by continued training for Ling 2.5 and 2.6, then trained Ling 3.0 from scratch with KDA and gated MLA at 5 : 1.
• NVIDIA's Nemotron models replace most attention with Mamba-2, a state-space layer that also carries a fixed-size state: only 6 of Nemotron 3 Nano's 52 layers are attention.

MiniMax later went the other way. M2 (Oct 2025) returned to full attention in all 62 layers, because hybrids that had matched full attention on standard benchmarks showed deficits in multi-hop reasoning at larger scale.

Side by side

Work counts multiply-adds per cached entry and head. MHA needs d_h for the score and d_h to mix values; absorbed MLA needs d_c + d_r for the score (latent plus rotary key) and d_c to mix latents.

Who uses it and why

"Given that sparsity 48 already offers strong performance, the marginal gains from doubling attention heads do not justify the inference cost."

"we empirically observed that learnable attention sinks bias dramatically enhance the performance of hybrid SWA models, matching or even surpassing baselines with fully GA layers."

"Long-context retrieval remains the primary bottleneck for pure linear attention, we therefore hybridize KDA with a small number of full global-attention (Full MLA) layers"

"The price paid became obvious at a larger scale: the model had clear deficits in complex, multi-hop reasoning tasks."

1. Vaswani et al., Attention Is All You Need (2017)
2. Shazeer, Fast Transformer Decoding: One Write-Head is All You Need (MQA, 2019)
3. Ainslie et al., GQA (2023)
4. DeepSeek-AI, DeepSeek-V2 (MLA, decoupled RoPE, 2024)
5. DeepSeek-AI, DeepSeek-V3 Technical Report and config.json
6. Kimi Team, Kimi K2 Technical Report
7. Z.ai, GLM-4.5 config.json
8. Gemma Team, Gemma 3 Technical Report
9. OpenAI, gpt-oss-120b & gpt-oss-20b Model Card
10. Xiaomi, MiMo-V2-Flash Technical Report and config.json
11. Xiao et al., Efficient Streaming Language Models with Attention Sinks (2023)
12. Yuan et al., Native Sparse Attention (2025), background for compressed and selected sparse branches
13. DeepSeek-AI, DeepSeek-V3.2-Exp and DeepSeek-V3.2 reports
14. DeepSeek-AI, DeepSeek-V4 Technical Report
15. Z.ai, GLM-5 Technical Report and GLM-5.2 blog (IndexShare)
16. Katharopoulos et al., Transformers are RNNs (2020)
17. Yang et al., Parallelizing Linear Transformers with the Delta Rule (DeltaNet, 2024)
18. Yang, Kautz and Hatamizadeh, Gated Delta Networks (2024)
19. Kimi Team, Kimi Linear Technical Report
20. Kimi Team, Kimi K3 Technical Report
21. Z.ai, GLM-5.3-Flash config.json
22. MiniMax, MiniMax-01 report (2025), MiniMax-M2 series report (2026), Why Did M2 End Up as a Full Attention Model?, MiniMax-M3 and M3.1-Flash model cards
23. Qwen, Qwen3-Next blog, Qwen3.8 model card and Qwen3.8-Next architecture report (2026)
24. StepFun, Step-3 report (2025), Step 3.5 Flash report (2026) and Inside Step 5 Preview
25. Ant, Ling and Ring 2.6 report (2026) and Ling-3.0-flash card; NVIDIA, Nemotron 3 Nano card; Mistral, Large 3 and Small 4 params.json

Lab Memory and compute calculator

How much memory does each attention design need for a long conversation, and how much does it read per generated token?

Pick models and a context length. The calculator uses each model's published configuration (hover a model name for its source) and the formulas from chapters 00 and 03. KV cache is what the model must keep in memory per sequence; reads per token is how many bytes of cache or state one generated token has to read. Numbers assume the cache is stored at the precision stated for the model, or 2 bytes (BF16) when none is stated; linear-attention states are also counted at 2 bytes, and at FP32 they would double.

04 Position information

How does attention know where each token sits, and why do several 2026 models leave position encoding out of their global layers?

Why attention needs position

An attention score depends only on the contents of a query and a key. Shuffle the earlier tokens and each query returns the same weighted sum, so "dog bites man" and "man bites dog" look alike. As the Transformer paper put it, "Since our model contains no recurrence and no convolution, in order for the model to make use of the order of the sequence, we must inject some information about the relative or absolute position of the tokens in the sequence." It added sinusoidal vectors to the input embeddings. Current models put position inside attention instead.

RoPE: rotate queries and keys

Draw two numbers of a query as an arrow in the plane. Rotary position embedding (RoPE), from RoFormer, turns that arrow by tθ for a token at position t, and turns a key at position s by sθ. A dot product is the two lengths times the cosine of the angle between the arrows. Turning changes no lengths, so the score changes only through that angle, which shifts by (s − t)θ: the distance between the tokens, not where they sit.

A toy with easy angles: θ = 20° per position, and a query and key that start 10° apart.

• Query at 2, key at 4. The query turns 40° and the key 80°, so they end 10° + 40° = 50° apart.
• Both shifted by 3, to 5 and 7. The query turns 100° and the key 140°: again 50° apart, same score.

Figure 4.1 draws both cases; its right panel shows that a real head turns many pairs, each at its own speed.

A head of dimension d_h splits each query and key into d_h/2 pairs and turns pair i by tθ_i, with

RoFormer uses base = 10,000, so pair 0 turns one radian per token and later pairs ever more slowly. Call the whole block-diagonal rotation R_t. Rotations add their angles and R_t^⊤ = R_−t, so, as in the toy, a query at t and a key at s score

Each vector carries its absolute position, but attention sees only the offset s − t. Values are not rotated.

YaRN calls λ_i = 2π/θ_i the wavelength: the number of tokens pair i needs for one full turn. In Figure 4.1's right panel, after 10 tokens pair 0 has turned 213° and pair 3 under 1°. Fast pairs separate neighboring tokens; slow pairs barely move over thousands of tokens. The base sets the slowest wavelength, which is why it is tied to context length.

def rope(x, pos, base=10_000.0, p=1.0):
    # x: [n, d_h] queries or keys of one head; pos: [n] positions t
    d_h = x.shape[-1]
    i = arange(d_h // 2)
    theta = base ** (-2 * i / d_h)            # [d_h/2], fastest pair first
    theta = where(i < int(p * d_h // 2), theta, 0.0)  # p-RoPE: drop slow pairs
    ang = pos[:, None] * theta[None, :]       # [n, d_h/2] angle t * theta_i
    x1, x2 = x[:, 0::2], x[:, 1::2]           # the d_h/2 pairs
    out = empty_like(x)
    out[:, 0::2] = x1 * cos(ang) - x2 * sin(ang)
    out[:, 1::2] = x1 * sin(ang) + x2 * cos(ang)
    return out                                # p = 1: RoPE; p = 0: no encoding

Rotating only part of each head

Many models rotate only some dimensions of each query and key. The rotated ones carry position; the unrotated ones score the same at any distance, so they compare content alone.

Why want such channels? Barbero et al. examined Gemma 7B and found it builds positional patterns from the fastest pairs and gives the slowest pairs the largest query and key norms, apparently using them as channels that compare content. Even slow pairs drift over long distances, so those channels are not robust. Their p-RoPE keeps the fraction p of pairs with the highest frequencies and stops rotating the rest: p = 0 is no encoding and p = 1 is RoPE. In Figure 4.1's toy head, 0.75-RoPE would freeze pair 3. Training 2B models at 8K, 0.75-RoPE reached validation perplexity 4.4414 on Wiki, against 4.4627 for RoPE and 4.8594 with no encoding.

Among the models here:

• Forced by MLA. In MLA (chapter 03), DeepSeek-V3's keys have 128 unrotated dimensions from the latent plus a 64-dimension rotated key.
• Chosen. GLM-4.5 sets partial_rotary_factor = 0.5; MiMo-V2-Flash rotates the first 64 of 192 query/key dimensions; DeepSeek-V4 rotates the last 64. Its cached entries serve as both keys and values, so the attention output would carry absolute position; V4 rotates the output back by the query's position, so that it "will also carry relative position embeddings".
• p-RoPE. Gemma 4 uses p = 0.25 on its global layers and RoPE on local ones, with bases of 1M and 10k.

No position encoding

A decoder can learn position with no encoding at all (NoPE). The causal mask gives each token a different set of visible tokens, and Kazemnejad et al. prove that the first layer can then recover absolute positions: a head that spreads attention evenly and reads a flag set only on the first token outputs 1/t at position t (at position 4 it averages four tokens, one flagged, and outputs 1/4). In their length-generalization tests, NoPE beat RoPE, ALiBi and absolute embeddings.

Several labs now drop RoPE on global layers and get order elsewhere (Figure 4.2):

• Local RoPE layers. Llama 4's iRoPE interleaves layers "without positional embeddings" among RoPE layers; in Meta's reference code the NoPE layers attend globally, the RoPE layers within local chunks, and at inference NoPE-layer queries are scaled up by a factor that grows with the log of position (beyond 8,192 tokens) to sharpen attention at long range. MAI-Thinking-1 uses RoPE (base 10,000) on its 512-token window layers and nothing on global ones.
• Linear-attention layers. Kimi Linear and Kimi K3 use NoPE in every MLA layer and let KDA's gated decay carry order and recency; GLM-5.3-Flash's config also turns RoPE off in its MLA layers (mla_use_nope).
• Mamba layers. NVIDIA's Mamba-2 hybrids, from Nemotron-H on, use no position encoding in any layer, because the Mamba layers provide implicit positional information.

The results cut both ways. In Kimi Linear's matched runs, NoPE beat a RoPE variant on long-context benchmarks, 54.5 vs 51.8 on average (RULER 84.3 vs 78.8), and without a rotary key, MLA also runs as pure MQA at inference. Qwen, however, tested NoPE for Qwen3.8-Flash-Next and kept partial RoPE: the two looked alike in pretraining, but the NoPE variant fell into endless generation far more often after post-training.

Learned relative positions: Inkling

Inkling takes a third route: instead of RoPE it learns a relative position embedding, citing Shaw et al. and the Music Transformer. It works like a table indexed by distance. In Shaw's form, each offset s − t, clipped to [−c, c], has its own learned vector a^K_ts, added to the key inside the score:

With c = 2 the table holds five vectors, and keys more than two tokens away share the edge one. Clipping also "enables the model to generalize to sequence lengths not seen during training". Hugging Face's write-up of Inkling describes a fourth projection, besides query, key and value, that produces "a per-token, per-head relative feature R", adjusted by the query-key distance and added into the attention logits. The exact formula has not been published.

Position and context extension

RoPE models are usually pretrained at shorter lengths and extended later (see the midtraining notes). The trouble is the slow pairs. If training stopped at length L, a pair whose wavelength exceeds L only ever saw part of a turn, and positions beyond L give it angles it never saw; the fast pairs have already turned many times, so every angle is familiar to them. In the base-10,000 toy head, 4,096 training tokens show pair 3 (λ = 6,283) about 0.65 of a turn and pair 0 about 650 turns. Four fixes are in use:

• Raise the base. A larger base leaves the fastest pair (θ₀ = 1) alone and slows the others, the slowest most, so over the longer context the slow pairs sweep roughly the angles they swept before, and neighboring tokens stay as easy to tell apart. YaRN describes this base change as "scaling high frequencies less and low frequencies more". MiMo-V2-Flash, for example, raised its global base 7.8× for an 8× longer context. A training stage at the new length then adapts the model to what changed.
• Position interpolation. Divide positions by a scale factor κ, squeezing the new range into the trained one: with κ = 4, position 10,000 gets the angles of position 2,500. κ is tuned, not set to the length ratio: Gemma 3 found κ = 8 to "work well" for a 4× extension from 32K to 128K.
• YaRN. Interpolate only pairs whose wavelength is long relative to the training length and leave fast pairs alone. YaRN also divides the logits by a temperature τ, fitted for lowest perplexity (√(1/τ) = 0.1 ln κ + 1), which it ties to attention becoming flatter (higher entropy) at extended lengths. DeepSeek-V3 applied YaRN only to the decoupled key with κ = 40 over a 4,096-token base, which covers 163,840 positions, a margin beyond the 128K it trained at; gpt-oss extended its full layers to 131,072 tokens with κ = 32 from 4,096.
• No encoding on global layers. NoPE global layers skip this step: Kimi K3 "extrapolates directly to 1M-token contexts without any positional-encoding modification".

Who uses it and why

"The global attention layers do not use any position encoding (Kazemnejad et al., 2023), which performs comparably to RoPE while being more efficient."

"Kimi K3 uses no explicit positional embedding (NoPE), and instead encodes positional information implicitly through the recurrent gating and decay mechanism of KDA."

"We find that encoding position with a relative positional embedding … performs better and extrapolates better to longer sequences than the more widely adopted Rotary Positional Embedding (RoPE)."

1. Vaswani et al., Attention Is All You Need (2017)
2. Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding (2021)
3. Peng et al., YaRN: Efficient Context Window Extension of Large Language Models (2023)
4. Barbero et al., Round and Round We Go! What makes Rotary Positional Encodings useful? (p-RoPE, 2024)
5. Kazemnejad et al., The Impact of Positional Encoding on Length Generalization in Transformers (NoPE, 2023)
6. Shaw et al., Self-Attention with Relative Position Representations (2018)
7. DeepSeek-AI, DeepSeek-V3 Technical Report and config.json
8. DeepSeek-AI, DeepSeek-V4 Technical Report
9. Z.ai, GLM-4.5 Technical Report, GLM-4.5 config.json, GLM-5.2 config.json, GLM-5.3-Flash config.json
10. Xiaomi, MiMo-V2-Flash Technical Report
11. Gemma Team, Gemma 3 Technical Report and Gemma 4 Technical Report
12. Meta, The Llama 4 herd (blog) and reference model.py
13. Microsoft AI, MAI-Thinking-1 Technical Report
14. Kimi Team, Kimi Linear Technical Report and Kimi K3 Technical Report
15. Thinking Machines, Introducing Inkling and Hugging Face write-up
16. OpenAI, gpt-oss Model Card and config.json
17. NVIDIA, Nemotron-H report and Nemotron 3 white paper; Qwen, Qwen3.8-Next architecture report

05 The residual stream and normalization

How does a 61-layer model like DeepSeek-V4-Pro keep activations and gradients at a usable size from the first layer to the last, and why did DeepSeek widen that path into four parallel streams?

The residual stream: a running sum

Picture each token carrying one vector of width d up through the network. Each sublayer (attention, then an MLP or MoE, alternating) reads the vector, computes an update and adds it back. Nothing is overwritten, so at the top the vector is the token's embedding plus every update written along the way. This running sum is the residual stream.

A toy with d = 2: the embedding (1, 0), plus attention's update (0, 2), plus the MLP's (1, −1), gives (2, 1). The MLP read (1, 2), which already held attention's write.

Why build it this way? Adding layers should never hurt a network, since the extra layers could learn to copy their input; in practice it did. He et al. called this the degradation problem: deeper plain networks reached higher training error, which rules out overfitting as the cause. Their shortcut lets each block learn only a correction F(x), and adds "neither extra parameter nor computational complexity". The unrolled sum shows why depth becomes trainable: the signal from layer l reaches layer L unchanged, and the gradient ∂x_L/∂x_l contains an identity term.

The rest of the chapter changes this picture: normalize what sublayers read (pre-norm) and what attention compares (QK-norm), widen the stream into several (HC, mHC, the Gated Residual), or weight earlier writes unequally (AttnRes).

Where the normalization goes

Each read passes through a normalization so the sublayer sees a predictable scale. Most of these models use RMSNorm (Zhang and Sennrich), which divides a vector by its root-mean-square size and multiplies by a learned gain g:

For x = (1, 7), RMS = √((1 + 49)/2) = 5, so with g = (1, 1) the output is (0.2, 1.4), the same as for (10, 70). RMSNorm drops LayerNorm's mean subtraction; it matched LayerNorm's quality and cut running time by 7% to 64%. Where the norm sits changes the picture:

• Post-LN, x_l+1 = Norm(x_l + F(x_l)), as in the original Transformer, normalizes the stream itself after each add. Xiong et al. showed that at initialization this gives large gradients to the parameters near the output, which is why Post-LN needs a learning-rate warm-up.
• Pre-LN, x_l+1 = x_l + F(Norm(x_l)), normalizes only the copy a sublayer reads and leaves the shortcut untouched. It has well-behaved gradients at initialization and trains without the warm-up. DeepSeek-V3, MiMo-7B and gpt-oss use RMSNorm here.
• Pre + post, x_l+1 = x_l + Norm_out(F(Norm_in(x_l))), also normalizes what the sublayer writes. Gemma 3, Gemma 4 and MAI-Thinking-1 use it.

Pre-LN's cost is that the stream is an unweighted sum whose size keeps growing with depth, which Kimi's Attention Residuals paper says ends up "progressively diluting each layer's relative contribution."

QK-norm: a norm inside attention

Pre-norm bounds the input to attention, but the logit q·k/√d_h also grows with W_q and W_k. When Dehghani et al. scaled vision transformers to about 8B parameters, training diverged because logits grew past 50,000 and attention became almost one-hot. QK-norm normalizes each head's queries and keys right before the dot product:

The logits then stay in a range set by the learned norm gains: with RMSNorm and gains of 1, query and key each have length √d_h, so a logit is at most √d_h, about 11.3 for d_h = 128. GLM-4.5, Gemma 3, Gemma 4, MAI-Thinking-1 and Muse Spark Open use QK-norm, and Llama 4 Scout applies a weightless version on its RoPE layers (Maverick's config turns it off). Qwen3 added QK-norm to ensure stable training, but in Qwen3-Next Qwen found some of those norm weights growing abnormally large and moved to zero-centered RMSNorm with weight decay on the norm weights. DeepSeek V4 normalizes each query head and its single compressed KV head. QK-norm needs explicit key vectors. In absorbed MLA (Chapter 03) the per-head keys are never formed at inference, so there is nothing to normalize; Chapter 10 shows how Kimi K2, which uses MLA, kept its logits bounded instead.

Hyper-connections: more than one stream

Hyper-connections (HC), from ByteDance, keep n_hc copies of the stream (the HC paper's "expansion rate"), a matrix X_l ∈ ℝ^n_hc×d, with three small maps per sublayer:

• Read. A_l ∈ ℝ^1×n_hc takes a weighted sum of the streams as the sublayer input.
• Write. C_l ∈ ℝ^n_hc×1 adds the output back to each stream with its own weight.
• Mix. B_l ∈ ℝ^n_hc×n_hc mixes the streams.

With n_hc = 1 and A = B = C = 1 this is the plain residual. F still sees one d-vector, so the sublayers are unchanged, and since n_hc is much smaller than d the extra FLOPs are small. In the dynamic version, A, B and C are predicted from the current token. With n_hc = 4, OLMoE-1B-7B with HC "converges 1.8 times faster" than without.

The trouble is depth. The signal from sublayer l reaches sublayer L through the product B_L−1 ⋯ B_l, and nothing keeps that product near the identity. Small gains compound: sixty mixes that each scale a signal by 1.1 scale it by 1.1⁶⁰ ≈ 304. In DeepSeek's 27B test model, HC's loss surged near step 12k along with the gradient norm, and the worst-case gain of the product (its largest absolute row or column sum) peaked near 3000. A plain residual has gain 1.

mHC: constrain the mixing matrix

DeepSeek's manifold-constrained hyper-connections (mHC) make every mix an average. Each B_l must be doubly stochastic: non-negative, with every row and every column summing to 1, so each output stream is a weighted average of the input streams. With two streams holding 4 and 0, the matrix with rows (0.75, 0.25) and (0.25, 0.75) gives 3 and 1: the total stays 4, and repeated mixing moves the streams toward 2 and 2 without ever growing them. The set of such matrices is called the Birkhoff polytope:

Such a matrix has spectral norm (largest singular value) at most 1, so mixing cannot amplify a signal in the forward or backward pass. Products of doubly stochastic matrices are doubly stochastic, so the bound holds at any depth. For n_hc = 1 the only such matrix is 1, the ordinary residual.

To put each predicted B on this set, mHC exponentiates it so every entry is positive, then runs the Sinkhorn–Knopp algorithm, dividing columns and then rows by their sums, for 20 iterations. A = σ(·) and C = 2σ(·) are kept positive so streams cannot cancel each other. C ranges over (0, 2), so a write weight of 1, the value a plain residual uses, sits in the middle. Twenty iterations give an approximate projection: each layer's own gain stays close to 1, and the gain of the product across all layers of the 27B model peaked at about 1.6 instead of 3000. It finished with 0.021 lower loss than the residual baseline, and BBH rose from 43.8 to 51.0. Figure 5.1 sets the two pictures side by side.

def mhc_sublayer(X, F, norm, phi, b, alpha, n_hc=4, t_max=20):
    # X: [n_hc, d] streams of one token
    # phi: [n_hc*d, n_hc**2 + 2*n_hc]; b: [n_hc**2 + 2*n_hc]
    x = X.reshape(-1)                                # [n_hc*d]
    h = (x / rms(x)) @ phi                           # [n_hc**2 + 2*n_hc]
    rd, wr = slice(0, n_hc), slice(n_hc, 2 * n_hc)   # read / write coefficient slots
    A = sigmoid(alpha[0] * h[rd] + b[rd])            # [n_hc] read weights
    C = 2 * sigmoid(alpha[1] * h[wr] + b[wr])        # [n_hc] write weights in (0, 2)
    M = exp(alpha[2] * h[2*n_hc:] + b[2*n_hc:]).reshape(n_hc, n_hc)
    for _ in range(t_max):                           # Sinkhorn-Knopp
        M = M / M.sum(axis=0, keepdims=True)         # columns sum to 1
        M = M / M.sum(axis=1, keepdims=True)         # rows sum to 1
    y = F(norm(A @ X))                               # [d]  one sublayer call
    return M @ X + C[:, None] * y[None, :]           # [n_hc, d]

DeepSeek-V4-Flash and V4-Pro use n_hc = 4 and 20 Sinkhorn iterations. DeepSeek-V4.1-Flash's "Single-Pass mHC" has each block use the read weights predicted by the previous block, so a fused deployment kernel reads and writes the residual once. Z.ai's GLM-5.3-Flash also uses mHC, with hc_mult = 4 and 20 Sinkhorn iterations in its config.

Reading the stream selectively: AttnRes and the Gated Residual

Kimi attacks the same equally weighted sum from another side. In the plain stream, layer l reads the sum of all earlier outputs, each with weight 1; with Attention Residuals (AttnRes) it learns the weights. Layer l holds one learned vector w_l ∈ ℝ^d, a "pseudo-query", and attends over the earlier layer outputs v_i: α_i→l = softmax_i(w_l^⊤ RMSNorm(v_i)) and h_l = ∑_i α_i→l v_i. The RMSNorm keeps layers with large outputs from dominating the weights.

Keeping every layer's output costs O(L·d) memory per token, so K3 sums the layers inside each block and attends only over block summaries: 8 blocks of up to 12 layers (the last one partial), 9 sources counting the embedding. In Kimi's scaling runs, Block AttnRes matched a baseline trained with 1.25× more compute, at a per-layer memory traffic of 5.5d against 34d for four-stream mHC.

Qwen's Qwen3.8-Flash-Next (Aug 2026) takes a third route. Its Gated Residual also widens the stream to four branches but has no mixing matrix B: each sublayer reads the branches through an elementwise, data-dependent sigmoid gate and writes back with one scalar per branch. Qwen credits the gate as a key contributor to the run's stability (chapter 10).

Who uses it and why

"the core idea of mHC is to constrain the residual mapping onto a specific manifold, and thus enhance the stability of signal propagation across layers while preserving model expressivity."

"each layer selectively retrieves representations from all preceding layers rather than accumulating them uniformly."

"We also incorporate QK-Norm [15] to stabilize the range of attention logits."

1. He, Zhang, Ren, Sun. Deep Residual Learning for Image Recognition (2015).
2. Xiong et al. On Layer Normalization in the Transformer Architecture (2020).
3. Zhang and Sennrich. Root Mean Square Layer Normalization (2019).
4. Dehghani et al. Scaling Vision Transformers to 22 Billion Parameters (2023).
5. Zhu et al. Hyper-Connections (ICLR 2025).
6. Xie et al. (DeepSeek-AI). mHC: Manifold-Constrained Hyper-Connections (v2, Jan 2026).
7. Kimi Team. Attention Residuals (2026).
8. DeepSeek-AI. DeepSeek-V4 technical report, §2.2, §3.4.2, §4.2.1; V4-Pro inference code (model.py).
9. DeepSeek-AI. DeepSeek-V4.1 technical report, §2.4.1.
10. Kimi Team. Kimi K3 technical report, §2.2.
11. Z.ai. GLM-5.3-Flash blog and config.json; GLM-4.5 report.
12. Gemma Team. Gemma 3 report; Gemma 4 report. Microsoft AI. MAI-Thinking-1 report. OpenAI. gpt-oss model card.
13. Qwen. Qwen3 report; Qwen3-Next blog; Qwen3.8-Next architecture report.

06 Multi-token prediction and draft models

How can a model emit several tokens per pass of its large network without changing its output, and why did labs in 2026 start training the network that proposes those tokens separately?

One token per pass

Language models are trained on next-token prediction: at every position t, minimize the cross-entropy of the true next token given all earlier tokens.

Generation mirrors this: 100 new tokens take 100 forward passes in sequence. Leviathan et al. observed that such inference is often limited by memory bandwidth: each pass reads all the weights and the KV cache to produce one token, and much of the compute sits idle (chapter 00). Two separate ideas put that idle compute to work:

• MTP as a training signal. Multi-token prediction (MTP) adds a loss for tokens further ahead. Decoding is unchanged, and the extra module can be thrown away.
• Speculative decoding. A cheap draft model guesses several tokens and the large model checks them all in one pass. Training is unchanged, and the output distribution stays exactly the same.

The two meet when a trained MTP module serves as the drafter.

Multi-token prediction as a training signal

Gloeckle et al. (2024) asked the model to predict the next few tokens at every position, with a shared trunk and one independent output head per future token. As an extra loss this helped at scale: their 13B models solved 12% more HumanEval and 17% more MBPP problems than next-token baselines, while small models got worse.

DeepSeek-V3 made the extra predictions sequential. Take "the cat sat on": at "cat" the main model predicts "sat", and the first MTP module, given the main model's hidden state at "cat" and the embedding of the true "sat", predicts "on". In general, the j-th module takes the hidden state h_t^j−1 at position t (from the main model when j = 1) and the embedding of token x_t+j, merges them, and runs one transformer block TRM_j to predict token x_t+j+1. Embedding and output head are shared with the main model. DeepSeek says this keeps "the complete causal chain at each prediction depth": because module j sees the actual token x_t+j, its prediction is conditioned on every earlier token x₁ … x_t+j, as the main model's is.

D_mtp is the number of extra tokens predicted (DeepSeek calls it the depth). V3 used D_mtp = 1 and loss weight λ = 0.3 for the first 10T tokens, then 0.1 for the last 4.8T. GLM-4.5 and MiMo-7B used the same 0.3 → 0.1 schedule. In V3's ablation on a 228.7B-parameter MoE, the module raised HumanEval from 44.5 to 53.7 at identical inference cost, since the module can be discarded after training.

Speculative decoding: draft, then verify

Speculative decoding pairs the large target model (next-token distribution p) with a cheap draft model (distribution q). One cycle has four steps:

1. The draft proposes γ tokens one at a time.
2. The target scores all γ + 1 positions in one pass, as it would a prompt.
3. Checking left to right, the target accepts each drafted token x with probability min(1, p(x)/q(x)) and stops at the first rejection.
4. The target samples one token itself: from the leftover distribution norm(max(0, p − q)) at the rejected position, or from its own p at the next position if all γ drafts were kept.

Each cycle yields between 1 and γ + 1 tokens, and the output is distributed exactly as p, however poor the draft. A toy with three words and one drafted token shows why. For each word the table gives q, p, the chance min(1, p/q) of keeping it if drafted, and its leftover max(0, p − q):

A drafted "the" is rejected one time in six, 0.6 × 1/6 = 0.1 of all cycles, and every rejection resamples "a", the only word with leftover mass. So "the" comes out 0.6 × 5/6 = 0.5 of the time, "a" 0.3 + 0.1 = 0.4 and "an" 0.1: exactly p. The draft survives 0.9 of the time, the overlap of the two distributions:

TV is total-variation distance (the paper calls it D_LK); the box gives the general proof.

Now chain γ drafts. Let α be the average β and assume acceptances are independent. Draft i is kept only if drafts 1 … i all are, with probability αⁱ, and the target always adds one token, so the expected yield is a geometric sum. For α = 0.9 and γ = 4 it is 1 + 0.9 + 0.81 + 0.729 + 0.656 ≈ 4.1.

The expected tokens per target pass is called the acceptance length. Let c be the cost of one draft step relative to one target pass, and assume verifying γ + 1 tokens costs about one ordinary pass, as it does while decoding is memory-bound. A cycle then costs γc + 1 passes:

With c = 0.1 the toy's 4.1 tokens take 1.4 passes, about 2.9× faster. Figure 6.1 draws such cycles to scale.

def speculative_step(target, draft, prefix, gamma):
    xs, qs = [], []
    for _ in range(gamma):                      # 1. draft gamma tokens
        q = draft.next_dist(prefix + xs)        # [V]
        xs.append(sample(q)); qs.append(q)
    ps = target.dists(prefix, xs)               # 2. one pass: [gamma+1, V]
    for i, x in enumerate(xs):                  # 3. verify left to right
        if uniform() < min(1.0, ps[i][x] / qs[i][x]):
            continue                            # accepted
        resid = maximum(ps[i] - qs[i], 0.0)     # rejected: resample from
        t = sample(resid / resid.sum())         # the leftover mass of p
        return prefix + xs[:i] + [t]
    return prefix + xs + [sample(ps[gamma])]    # 4. all kept: bonus token

MTP modules as drafters

Here the two ideas meet. A trained MTP module is a ready-made draft model: small, fed by the target's hidden state, and trained to predict ahead. DeepSeek-V3's single module reached 85–90% acceptance for the second token and 1.8× tokens per second. MiMo-7B fine-tuned two extra MTP layers after pretraining; the first layer accepted about 90% and the third above 75%.

Drafting several steps from one module creates a mismatch: training feeds each step the target's hidden states, inference feeds later steps the module's own outputs. GLM-5 trained 3 MTP steps with shared weights to close that gap and reported an acceptance length of 2.76 against 2.55 for DeepSeek-V3.2 at 4 speculative steps. GLM-5.2 changed three more things:

• Shared state. It shared the KV cache and sparse-attention indices across steps.
• A loss on acceptance. It trained with an end-to-end TV loss; since β = 1 − TV, that loss targets acceptance directly.
• Rejection sampling. It adopted the min(1, p/q) test above, following Bebop (2026), which contrasts it with a common "target-only" check: the draft proposes its single most likely token ŷ and the target keeps it with probability p(ŷ).

In an ablation on the GLM-5.1 backbone with 7 MTP steps in training and inference, acceptance length on coding rose from 4.56 to 5.47.

2026: drafters trained on their own

During 2026 several labs took the drafter out of backbone pretraining or retrained it afterwards. A separate drafter no longer shapes backbone pretraining, can be retrained whenever the policy changes (which matters for RL, where generating rollouts is the bottleneck), and can draft many tokens per pass while training on acceptance itself.

• DeepSeek V4.1-Flash dropped the MTP module from backbone pretraining. Its DSpark drafter, three transformer blocks that compute five draft positions in one pass, trains in its own stage with the backbone frozen, then keeps training through post-training (RL and on-policy distillation, OPD) without sending gradients into the backbone. A confidence head predicts which drafts will survive, and a scheduler uses it to pick how many to verify per request.
• Xiaomi MiMo-V2.6 keeps an MTP objective in pretraining but drafts with a 5-layer DFlash block-diffusion model. Block diffusion here means the drafter takes a block of masked future positions after one known anchor token, lets them attend to each other in both directions, and fills the whole block in one forward pass. MiMo's version predicts 7 tokens in one forward pass. For RL rollouts, a variant with draft block size 6, fine-tuned on rollout logs, gave 31.3% longer accepted lengths than the MTP-3 setup it replaced.
• Gemma 4's drafters ship as separate "-assistant" checkpoints: 4-layer transformers that cross-attend to the target's KV cache, so they need no prefill. Google reports up to 3× faster decoding.
• Kimi K3 fine-tunes its pretrained MTP layer into an EAGLE-3 draft model with the target frozen. Training unrolls the draft for seven steps and from the second step on feeds it its own earlier predictions, as happens at inference (EAGLE-3 calls this "training-time test"). It minimizes the LK loss −log ∑_x min(p(x), q(x)), the negative log of the acceptance rate.
• Meta trained Muse Glimmer's DFlash drafter offline on the target's own rollouts; it decodes 3.1× faster on an RTX 5090.

The recipe descends from EAGLE (2024), a one-decoder-layer draft head on the frozen target's features that gave 2.7×–3.5× speedups on LLaMA2-Chat 70B.

Who uses it and why

"The training-inference discrepancy reduces the acceptance rate of the second token. Therefore, we propose sharing the parameters of 3 MTP layers during training."

"MTP acceleration is particularly well-suited for RL training (RadixArk Team, 2025), where the rollout phase consistently emerges as the dominant bottleneck"

"This keeps DSpark aligned with the evolving policy, enabling it to accelerate both online serving and rollout generation for RL and OPD."

1. Gloeckle et al. Better & Faster Large Language Models via Multi-token Prediction (2024)
2. Leviathan, Kalman, Matias. Fast Inference from Transformers via Speculative Decoding (2023)
3. Li et al. EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty (2024)
4. Li et al. EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test (2025)
5. DeepSeek-AI. DeepSeek-V3 Technical Report (2024)
6. DeepSeek-AI. DeepSeek-V4.1-Flash Technical Report (2026)
7. Z.ai. GLM-5 Technical Report (2026) and GLM-5.2 blog
8. Xiaomi. MiMo-7B report (2025), MiMo-V2-Flash report (2025) and MiMo-V2.6 report (2026)
9. Gemma Team. Gemma 4 report (2026) and gemma-4-31B-it-assistant model card
10. Kimi Team. Kimi K3: Open Frontier Intelligence (2026) and Moonlight report (2025)
11. Meta. Muse Glimmer 30B report (2026) and model card
12. Qwen, Qwen3.5 blog; MiniMax, MiniMax-M3 model card

07 Multimodal input

How does a language model take in a photo or a sound clip, and why are labs replacing attached vision encoders with raw pixel patches fed straight into the model?

Pixels and sound as vectors

A transformer reads a sequence of vectors of width d; for text they come from an embedding table. The Vision Transformer (ViT) turns an image into such a sequence in three moves:

1. cut the image into square patches of side p pixels;
2. flatten each into p²·C numbers (C = 3 colors);
3. map each to the encoder width d_v with one learned matrix W_patch, and add a learned position vector.

A 28×28 image with p = 14 gives a 2×2 grid of patches of 14 · 14 · 3 = 588 numbers each. In general an H×W image gives P = HW / p² patches, the rows of X:

Transformer layers of width d_v follow. Vectors like these that enter a language model (LM) are soft tokens (Gemma 3's term), written "at the corresponding image-token positions in the input embedding sequence" (DeepSeek-V4.1-Flash).

Audio is cut along time. Gemma 4 12B slices 16 kHz audio into 40 ms chunks of 16,000 × 0.04 = 640 samples. Other systems encode a log-mel spectrogram, the energy in each frequency band of each short frame. dMel skips the encoder: it rounds each of 80 mel energies to one of 16 levels, a discrete code per frame.

Three routes into the model

Every design ends with soft tokens in the LM's input. The designs differ in what makes those vectors and when the LM first trains on them:

• Adapter on a frozen encoder. A small projector is trained between a separately pretrained, frozen encoder and a finished text model.
• Continued pretraining. The encoder is attached to a text checkpoint, and the whole model keeps training on mixed data.
• Native early fusion. Images are in the mix from the LM's first step, through an encoder or through one light projection that leaves the visual processing to the LM.

Figure 7.1 draws the routes side by side, splitting the native route by whether an encoder remains; the bar under each column marks when image data joins training.

Adapter: a pretrained encoder plus a projector

Raw pixels through one matrix are a weak start, so early vision-language models borrowed an encoder that had already learned to see. Contrastive training gives matching image-caption pairs similar vectors. SigLIP replaces CLIP's softmax over the whole batch with a sigmoid on each pair, which "does not require a global view of the pairwise similarities for normalization".

LLaVA (2023) set the baseline: a frozen CLIP ViT-L/14 and one trainable matrix W into the LM's word-embedding space. Stage 1 trains only W, "a compatible visual tokenizer for the frozen LLM"; stage 2 also tunes the LM.

Continued pretraining: vision added to a text checkpoint

Next, labs attached the encoder to a text checkpoint and trained the whole model on mixed data for trillions of tokens:

• Kimi-VL added 2.3T joint tokens to a Moonlight checkpoint trained on 5.2T text tokens.
• Kimi K2.5 added 15T vision-text tokens to a near-end K2 checkpoint, with a 400M SigLIP-initialized MoonViT-3D.
• MiMo-V2.6-Flash trained 26T text tokens, then 22T omni-modal tokens with its 681M ViT and audio encoder attached.
• GLM-4.5V trained an AIMv2-initialized ViT, an MLP adapter and GLM-4.5-Air together for 120,000 steps.

This reuses the costly text model but abruptly shifts its input distribution. With the total vision-text budget fixed, Kimi found that adding vision from 0% of training at a 10:90 ratio beat adding it at 80% with 50:50 on all six vision and text scores (text knowledge 45.5 against 43.1), and that later injection caused a "dip-and-recover" in text ability. K2.5 therefore mixes vision at a constant ratio.

Native multimodal pretraining

Native models see images from the LM's first step. Several labs now train this way:

• Llama 4's early fusion puts text and vision tokens into one backbone, with a MetaCLIP-based encoder trained separately against a frozen Llama.
• Gemma 3 mixes images in through its frozen encoder.
• GLM-5.3-Flash, "the first natively multimodal model in the GLM-5 series", uses a 30T-token multimodal corpus.
• Qwen3.5 (Feb 2026) brought vision into its flagship base through early text-vision fusion, and MiniMax M3 (Jun 2026) mixed text, image and video from the first step.

DeepSeek-V4.1-Flash prepares its encoder in steps:

1. pretrain the ViT apart, with SigLIP's loss on about 47B pairs;
2. continue it with next-token training on 236B tokens through a throwaway 4B MoE LM;
3. keep it frozen in the main run until learning-rate decay, while that run mixes text-only and multimodal tokens 7:1 over 45T tokens.

The main run also gives each modality its own MoE load-balancing biases, because image and text tokens may prefer different experts.

Kimi K3 trains its 401M MoonViT-V2 from scratch with next-token prediction. In Kimi's ablation, a SigLIP-initialized encoder made joint training unstable: its gradient norms stayed higher and spiked often. Kimi also argues that a contrastive loss favors global meaning over fine text and structure. The from-scratch encoder matched the SigLIP-initialized one on vision evaluations.

Encoder-free: patches straight into the language model

Encoder-free designs keep only a light patch embedding and let the LM's own layers do the perception. Gemma 4 12B maps each 48×48×3 RGB patch (6,912 numbers) through one 35M-parameter matmul in place of a 550M ViT, then adds 2D coordinate embeddings and a LayerNorm. Its 305M audio encoder is gone too: each raw 40 ms chunk is projected directly.

Inkling feeds audio as dMel spectrograms and images as 40×40-pixel patches through a four-layer hMLP. This hierarchical MLP stem (Touvron et al.) merges sub-patches step by step with linear layers and never mixes neighboring patches, like a convolution whose kernel equals its stride. MAI-Base-2 is also encoder-free; vision-language data made up 18% of its 50T-token mix, from 4.5T unique tokens.

Google cites latency, memory use and memory fragmentation, and notes the 12B can be fine-tuned in one pass. Thinking Machines chose it to fit its interaction models, which let users collaborate "using voice and vision in real time".

What an image costs in tokens

Every vector costs the LM one context position, so most designs merge neighboring patches first:

• Pooling. Gemma 3's frozen 400M SigLIP encoder takes 896×896 inputs and average-pools 4×4 to 256 tokens. Since the encoder never changes, Google pre-computes the embeddings, adding no encoder cost to training.
• Pixel shuffle. Kimi-VL stacks each 2×2 block of features into one four-times-wider vector before a two-layer MLP.

With patch side p and merge factor r, each LM token covers an rp × rp square of pixels. At p = 14 and r = 2 that is one token per 28² = 784 pixels, so a 448×448 image costs (448/28)² = 16² = 256 tokens. In shapes, for an encoder of width d_v and an LM of width d:

def patchify(img, p):              # img (H, W, C); p divides H, W
    H, W, C = img.shape
    x = img.reshape(H//p, p, W//p, p, C).transpose(0, 2, 1, 3, 4)
    return x.reshape(H//p, W//p, p * p * C)    # flattened patches

def unshuffle(g, r):                # r x r neighbours -> channels
    h, w, c = g.shape
    g = g.reshape(h//r, r, w//r, r, c).transpose(0, 2, 1, 3, 4)
    return g.reshape(h//r, w//r, r * r * c)

def encoder_tokens(img, vit, mlp, p=14, r=3):    # DeepSeek-V4.1
    f = vit(patchify(img, p))                   # (H/p, W/p, 1024)
    return mlp(unshuffle(f, r)).reshape(-1, d)   # (HW/(rp)^2, d)

def encoder_free_tokens(img, W_in, pos, p=48):   # Gemma 4 12B
    x = patchify(img, p) @ W_in                  # 6912 -> d
    return layer_norm(x + pos).reshape(-1, d)

def audio_tokens(wave, W_a, n=640):   # 40 ms of 16 kHz per token
    frames = wave[: len(wave) // n * n].reshape(-1, n)
    return frames @ W_a                          # 25 tokens/s

x = embed(ids)                                   # text rows
x[image_slots] = encoder_tokens(img, vit, mlp)   # soft tokens

Who uses it and why

"Early fusion with a lower vision ratio yields better results given a fixed total vision-text token budget."

"We depart from this practice primarily for training stability. When a pre-trained encoder is attached to the LLM, joint optimization becomes unstable"

"Because these split encoders add latency and increase memory usage, we trained Gemma 4 12B with an encoder-free architecture to integrate audio and vision input directly."

1. Dosovitskiy et al., An Image is Worth 16x16 Words (ViT), 2020
2. Zhai et al., Sigmoid Loss for Language Image Pre-Training (SigLIP), 2023
3. Liu et al., Visual Instruction Tuning (LLaVA), 2023
4. Touvron et al., Three things everyone should know about Vision Transformers (hMLP stem), 2022
5. Bai et al., dMel: Speech Tokenization made Simple, 2024
6. Gemma Team, Gemma 3 Technical Report
7. Gemma Team, Gemma 4 Technical Report; Gemma 4 12B blog; model card; processor_config.json
8. Kimi Team, Kimi-VL Technical Report
9. Kimi Team, Kimi K2.5 Technical Report; model card
10. Kimi Team, Kimi K3 Technical Report; model card
11. DeepSeek-AI, DeepSeek-V4.1-Flash Technical Report
12. Xiaomi MiMo, MiMo-V2.6 Technical Report; MiMo-V2.5 blog
13. GLM-V Team, GLM-4.1V-Thinking and GLM-4.5V; GLM-5.3-Flash blog
14. Meta, The Llama 4 herd
15. Thinking Machines, Introducing Inkling
16. Microsoft AI, Frontier Pre-training at Microsoft AI (MAI-Base-2)
17. StepFun. Step 3.5 Flash report. Qwen. Qwen3.8-Next architecture report. NVIDIA. Nemotron 3 Nano report. Ant. Ling 2.0 report. MiniMax. MiniMax-01 report.

08 Optimizers: AdamW and Muon

What does Muon do to a weight matrix's update that AdamW does not, and why did seven of the eleven labs that name their pretraining optimizer move their weight matrices to it by October 2026?

From SGD to Adam: a step size for every weight

Stochastic gradient descent moves each weight by −η times its gradient, where η is the learning rate. Momentum keeps a running sum of past gradients, M_t = μM_t−1 + G_t, and steps along that instead, which smooths out batch-to-batch noise. One learning rate η must then suit every weight, whether its gradients are large or small.

Adam (Kingma and Ba) sets the scale per coordinate. For each weight it keeps a running mean of the gradient, m, and of the squared gradient, v, and steps by about η · m/√v. Multiplying every gradient by 1,000 leaves that ratio unchanged; what it measures is how consistent the gradient has been:

• Steady gradient. A gradient of 0.001 at every step drives m and √v both to 0.001: a step of η. A steady gradient of 10 gives the same step.
• Noisy gradient. A gradient flipping between +1 and −1 leaves m near ±0.05 (with β₁ = 0.9) and √v at 1: a step of about 0.05η.

In full, β₁ and β₂ set how long each average remembers, and the hats correct both averages for starting at zero:

The paper's defaults are β₁ = 0.9 and β₂ = 0.999; DeepSeek-V3 and MiMo use a shorter memory for v, β₂ = 0.95. Adam stores two extra numbers per parameter.

The λθ term is weight decay: each step also shrinks every weight by the fraction ηλ. Libraries used to implement it as an L2 penalty added to the gradient. Loshchilov and Hutter showed that inside Adam that penalty also gets divided by √v̂, so weights with large past gradients are regularized less. AdamW applies the decay directly to the weights, outside the adaptive scaling, as in the equation above. DeepSeek-V3, MiMo-7B and MAI-Thinking-1 all use λ = 0.1 (MAI lowers it for attention and embedding weights).

Muon: equalize the directions of a matrix update

Adam treats a d×d weight matrix as d² unrelated numbers. Keller Jordan observed that the updates SGD-momentum and Adam produce for transformer weight matrices "typically have very high condition number": they are close to low-rank, dominated by a few directions. Muon (MomentUm Orthogonalized by Newton–Schulz) works on the whole matrix: it keeps SGD momentum and replaces the momentum matrix by the nearest semi-orthogonal matrix (orthonormal rows or columns, so all its singular values equal 1).

A 2×2 example shows what that does:

The columns of G, (3, 4) and (−0.4, 0.3), are 5 × (0.6, 0.8) and 0.5 × (−0.8, 0.6). They are perpendicular, so these are G's singular directions, with singular values 5 and 0.5. Orthogonalizing keeps both directions and sets both strengths to 1. The step rescales O anyway (by η and, below, a shape factor), so what matters is the ratio of strengths: it falls from 10 to 1, and the weak direction now moves as far as the strong one (Figure 8.1).

In general, the singular value decomposition M = UΣV^⊤ splits a matrix into directions (U, V) and strengths (the singular values in Σ). Muon's update is UV^⊤: every singular value becomes 1, so every direction moves by the same amount, including "rare directions" that Jordan suspects matter for learning.

In practice Muon orthogonalizes the Nesterov form μM_t + G_t instead of M_t: it looks one momentum step ahead, which Jordan found works a bit better in every case he tested, and Moonlight and DeepSeek V4 do the same. The pseudocode below follows that form.

Newton–Schulz: orthogonalizing with matrix multiplies

An SVD every step would be too slow. A Newton–Schulz iteration reaches nearly the same UV^⊤ without computing U or V, by repeating one step built from matrix multiplies:

Because XX^⊤ = UΣ²U^⊤, one step applies the odd polynomial φ to every singular value and leaves U and V alone. Two choices make φ pull every singular value toward 1:

• Start at or below 1. The iteration begins from X₀ = M/‖M‖_F, which puts every singular value at or below 1, because the Frobenius norm is at least the largest singular value. For σ well above 1 the cσ⁵ term dominates and the polynomial blows up.
• Grow small values fast. Jordan tuned (a, b, c) = (3.4445, −4.7750, 2.0315) to make a, the slope at zero, as large as possible, so a small singular value grows about 3.4 times per step. The price is that φ does not settle at exactly 1: the values keep oscillating in a band Jordan puts at [0.7, 1.3], which he found does not hurt the loss, and 5 steps sufficed in his experiments.

The iteration runs stably in bfloat16 and costs little. For an A×B weight with B ≤ A, each of T steps costs at most about 6AB² FLOPs, while the forward and backward pass costs about 6AB per token, so Jordan bounds Muon's extra FLOPs, as a fraction of the training FLOPs, by T·B / (tokens per batch). That is about 0.5% for a Llama-405B-sized run. Figure 8.2 follows the singular values of a 3×3 example, worked through below.

def newton_schulz(M, steps=5, eps=1e-7):
    a, b, c = 3.4445, -4.7750, 2.0315        # Jordan's quintic coefficients
    X = M / (frobenius_norm(M) + eps)        # all singular values <= 1
    tall = X.shape[0] > X.shape[1]
    if tall:
        X = X.T                              # keep X @ X.T the smaller Gram matrix
    for _ in range(steps):
        A = X @ X.T
        X = a * X + (b * A + c * A @ A) @ X  # sigma -> a*s + b*s^3 + c*s^5
    return X.T if tall else X

def muon_step(W, G, M, lr, mu=0.95, wd=0.1, rms=0.2):
    # W, G, M: [A, B] weight, gradient, momentum buffer
    M = mu * M + G                           # SGD momentum
    O = newton_schulz(mu * M + G)            # Nesterov look-ahead, then orthogonalize
    O = O * rms * sqrt(max(W.shape))         # update RMS ~ rms, as for AdamW (Moonlight)
    W = W * (1 - lr * wd) - lr * O           # decoupled weight decay, then step
    return W, M

Which parameters stay on AdamW

Muon is defined for 2D weight matrices, so scalars and vectors such as RMSNorm gains and biases stay on AdamW. Jordan also found that the embedding and the output head, though 2D, train best with AdamW. Moonlight keeps "RMSNorm, LM head, and embedding parameters" on AdamW, DeepSeek V4 adds mHC's static biases and gating factors to that list, and MAI-Base-2 runs Muon on its "non-embedding 2D parameters". GLM-4.5 lists only word embeddings, biases and RMSNorm weights as non-Muon parameters. DeepSeek V4.1-Flash adds Engram, a conditional memory: huge embedding tables indexed by hashes of the last 2 to 4 tokens, whose looked-up vectors are gated into the hidden state. It trains those tables, its embeddings and its prediction head with a "Sinkhorn-balanced" momentum update instead of Adam, which needs one state buffer instead of Adam's two. Qwen3.8-Flash-Next has a similar hashed memory, 51B parameters of n-gram embedding tables, trained with Adam without weight decay, and keeps its embeddings, output head, MoE router and gates on AdamW.

Moonlight: making Muon work at scale

Muon set speed records on small models. Kimi's Moonlight report found two problems at scale:

• Weights grow. Without weight decay the RMS of the weights and layer outputs kept growing, "exceeding the high-precision range of bf16". Adding AdamW-style decoupled weight decay fixed it.
• Update size depends on shape. For a full-rank A×B matrix, UV^⊤ has min(A, B) singular values equal to 1, so its squared Frobenius norm is min(A, B), spread over A·B entries: its RMS is √(min(A, B)/(A·B)) = 1/√max(A, B). A 100×400 matrix gets an update RMS of 1/20 = 0.05 and a 100×100 matrix 1/10 = 0.1. Wide MLP matrices get tiny updates and small per-head matrices get large ones.

Moonlight multiplies each update by 0.2·√max(A, B), which puts every update's RMS near 0.2 (for the 100×400 matrix, 0.05 × 0.2 × 20 = 0.2), inside the 0.2 to 0.4 range AdamW's updates already have:

Muon can then reuse the learning rate and weight decay tuned for AdamW. In Moonlight's scaling-law fits, Muon matched AdamW with about 52% of the training FLOPs, which the paper calls "∼2× computational efficiency". Kimi K2 calls this recipe plus QK-Clip (Chapter 10) MuonClip.

The recipe spread:

• GLM-4.5 scales updates to RMS 0.2 with 5 Newton–Schulz steps.
• DeepSeek V4 rescales to 0.18 with momentum 0.95 and weight decay 0.1.
• StepFun and Qwen orthogonalize with Polar Express, a Newton–Schulz variant: Step 3.5 Flash runs 6 steps; Qwen3.8-Flash-Next keeps Moonlight's 0.2 scaling and runs 8, which Qwen found more accurate and less prone to gradient-norm spikes.

Per-head Muon

Jordan's post had already reported that Muon works better on Q, K and V separately than on one fused QKV matrix. The 2026 models go one level further. A projection such as W_Q stacks all heads in one matrix, and orthogonalizing it as a whole couples them. In Kimi K3's words, heads with larger gradient or momentum scales "dominate the shared update direction, while smaller-scale heads receive insufficiently normalized updates". Splitting the momentum along the head dimension and orthogonalizing each head's block separately gives every head the same update scale.

Three labs adopted versions of this within seven months. GLM-5's Muon Split orthogonalizes the per-head slices of the MLA up-projections, which closed MLA's gap to GQA in their ablation and kept attention-logit scale stable without any clipping. Kimi K3's Per-Head Muon does it for Q, K and V, and notes that the tall per-head blocks also make Newton–Schulz slightly cheaper. DeepSeek V4.1-Flash's head-wise Muon splits the Query and Key weights; DeepSeek describes it as one preconditioner per head instead of one for all heads, and found that it "outperforms vanilla Muon".

Adoption

Of the eleven labs that state a pretraining optimizer, seven moved their weight matrices to Muon:

• Muon. Kimi first, then Z.ai (GLM), StepFun (Step 3.5 Flash, Feb 2026), DeepSeek, Thinking Machines (Inkling), Qwen (Qwen3.8-Flash-Next, Aug 2026, a preview of its next architecture; Qwen3.5 used AdamW) and Microsoft AI (MAI-Base-2).
• AdamW, then Muon. Xiaomi still pretrains MiMo-V2.6 with AdamW, then switches the hidden matrices to Muown, a Muon variant, in mid-training, citing Muon's data efficiency for large-batch RL.
• AdamW. NVIDIA (Nemotron 3) and Ant (Ling 2.0) pretrain with AdamW, and MiniMax last named an optimizer, AdamW, for MiniMax-Text-01.

Figure 8.3 places the main releases on a timeline.

Who uses it and why

"We employ the Muon (Jordan et al., 2024; Liu et al., 2025) optimizer for the majority of modules in DeepSeek-V4 series due to its faster convergence and improved training stability."

"When scaling up to MAI-Base-2, we found that Muon delivered more stable training dynamics and large efficiency gains over AdamW."

"per-head orthogonalization equalizes the update scale across heads."

1. Kingma and Ba. Adam: A Method for Stochastic Optimization (2014).
2. Loshchilov and Hutter. Decoupled Weight Decay Regularization (2017).
3. Jordan et al. Muon: An optimizer for hidden layers in neural networks (blog post, Dec 2024).
4. Liu et al. (Moonshot AI). Muon is Scalable for LLM Training (Moonlight) (2025).
5. Kimi Team. Kimi K2 technical report, §2.1; Kimi K3 technical report, §2.5, §3.3.
6. Z.ai. GLM-4.5 report, §2.4; GLM-5 report, §2.1.
7. DeepSeek-AI. DeepSeek-V4 report, §2.4, §4.2.2; DeepSeek-V4.1 report, §2.5; DeepSeek-V3 report, §4.2.
8. Microsoft AI. Frontier Pre-training at Microsoft AI (MAI-Base-2); MAI-Thinking-1 report, §2.6.2.
9. Xiaomi. MiMo-V2.6 report, §3; MiMo-7B report; MiMo-V2-Flash report.
10. Thinking Machines. Introducing Inkling.
11. NVIDIA. Nemotron 3 Super report. Ant. Ling 2.0 report. MiniMax. MiniMax-01 report. Qwen. Qwen3.8-Next architecture report.

09 Learning-rate schedules and hyperparameter scaling

How do labs set the learning rate at every step of a trillion-token run, and the batch size and weight decay that go with it, when the final model can be trained only once?

The learning rate η scales every optimizer update. A large η makes fast progress but leaves the weights bouncing in a noisy region; a small η lets them settle. A schedule plans when to do which.

The shapes at a glance

A schedule is assembled from a few pieces (Figure 9.1). Warmup comes first. After it, the shapes differ in when they commit to the run's length:

• Cosine decays over the whole run, so it needs the final step from the start.
• Warmup-stable-decay (WSD) holds η constant and decays only at the end, so the end can be chosen late. A minus-square-root decay is a WSD decay that drops fast at first and then flattens.
• Checkpoint merging never decays: η stays constant to the end, and averaging the weights of the last few checkpoints takes the decay's place.

Warmup

Warmup raises η linearly from zero over the first steps; the original Transformer used 4,000. Xiong et al. traced the need to Post-LN blocks (normalization after the residual add), whose gradients near the output are large at initialization, and showed that Pre-LN models can train without it. Labs still warm up: DeepSeek-V3 for 2K steps, Kimi K2 for 500 steps, Kimi K3 for 1% of training, MAI-Base-1 for about 12B tokens.

Cosine decay

The cosine schedule then lowers η along half a cosine wave from the peak η_max to a floor η_min at the final step S (s is the current step, W the last warmup step):

S must be the true run length. Chinchilla got the best final loss when the cosine length matched the token count, and overshooting it by more than 25% noticeably hurt. A cosine run commits to its budget on day one: mid-run checkpoints still have high η, and training longer means re-warming η (as Moonlight did for its last 0.5T tokens, after a full cosine) or starting over.

Warmup-stable-decay

MiniCPM split training into a long phase at constant η and a short decay, the warmup-stable-decay (WSD) schedule. With T the end of the stable stage and 0 < f ≤ 1 any decreasing function:

On a 0.036B model, the loss dropped sharply once decay began and reached or beat the cosine run of the same length. Decaying over 10% of total tokens was enough; 2.5% fell short.

The stable phase does not need to know when training ends, so any stable checkpoint can be branched: decay a copy to get a finished model and keep training the original. With a 10% decay, one stable run yields a 6T-token model (5.4T stable tokens, then 0.6T of decay on a copy) and later a 10T-token model (resume to 9T, decay 1T), for 10.6T tokens of compute instead of 16T for two separate runs. MiniCPM measured data scaling this way, with one stable run per model size and decayed branches at several token counts. For m model sizes × m data sizes, this cuts the work from about m² training runs to about m. A lab can also extend a run by resuming from the last checkpoint before decay.

def lr_cosine(s, W, S, peak, floor):
    if s < W:
        return peak * s / W                       # linear warmup
    p = (s - W) / (S - W)                         # 0 -> 1, needs the final step S
    return floor + 0.5 * (peak - floor) * (1 + cos(pi * p))

def lr_wsd(s, W, T, S, peak, floor):
    if s < W:
        return peak * s / W
    if s < T:
        return peak                               # stable: no end date needed
    p = (s - T) / (S - T)                         # short decay, e.g. S - T = 0.1 * S
    return floor + 0.5 * (peak - floor) * (1 + cos(pi * p))  # cosine-shaped decay

# WSD branching: one stable run yields finished models at 6T and 10T tokens
ckpt = train(init(), lr=peak, tokens=5.4e12)      # stable phase
model_6T = decay(copy(ckpt), tokens=0.6e12)       # branch off and decay 10%
ckpt = train(ckpt, lr=peak, tokens=3.6e12)        # resume the stable run to 9T
model_10T = decay(ckpt, tokens=1.0e12)            # total compute: 10.6T tokens

Figure 9.2 puts cosine and WSD on one run, with a branch, and then draws DeepSeek-V3's published schedule to scale.

Who uses which

DeepSeek, Kimi K2 and MiMo use a WSD shape whose decay follows a cosine:

GLM-4.5 chose plain cosine from 2.5×10⁻⁴ to 2.5×10⁻⁵ because its WSD runs scored worse on SimpleQA and MMLU, which it read as underfitting in the stable stage. MAI-Base-1 decays along a cosine from 2×10⁻⁴ to 2×10⁻⁵, a 0.1× floor rather than the common 0.01×, because decaying less helped after RL. Kimi used WSD for K2 and moved to cosine for K3 after tuning each schedule separately, since their optimal peak η and batch sizes differ:

"Under their respective optimal hyperparameter settings, cosine decay consistently achieves a lower final loss than WSD."

Two of the labs the atlas added change how the run ends. NVIDIA's Nemotron 3 Super holds its peak for 20T of 25T tokens, then decays along a minus-square-root curve over the last 5T, and the base it hands to post-training is a merge of checkpoints over a 500B-token window. Ant drops the decay entirely: Ling 2.0's warmup-stable-merge (WSM) schedule keeps η constant to the end and gets the annealing effect by merging checkpoints, which for Ling-1T come from its mid-training.

Gemma, gpt-oss, Llama 4, Inkling and Mistral do not publish their schedule shapes.

Batch size moves too

The batch size B interacts with η, and MiniCPM found that the optimal batch size grows as the loss falls. Labs either ramp B up early in the run or hold it fixed:

• Ramp. DeepSeek-V3 ramps B from 3,072 to 15,360 over the first 469B tokens; the report gives no unit. If each unit were one sequence of the 4K pre-training length, that would be 12.6M to 62.9M tokens, but this is not stated. GLM-4.5 ramps from 16M to 64M tokens over the first 500B. MiniMax-Text-01 ramped furthest, doubling its batch three times, from 16M to 128M tokens by 4.7T, at points set by a fitted power law between loss and critical batch size.
• Fixed. Kimi K2 holds 67M tokens and DeepSeek V4.1-Flash 100.6M tokens throughout. Qwen dropped the ramp for Qwen3.8-Flash-Next after a test in which it cost 18.8% more steps for no gain.

Carrying hyperparameters up in scale

The best η, B and weight decay λ shift with model size and token count, and no lab can grid-search them on the final model. Labs take one of two routes:

• A parameterization under which the optimum stays put. MiniCPM used Tensor Programs (μP), which multiply each weight matrix's learning rate by d_base/d for a small reference width d_base, so a model 8 times wider than the reference trains its matrices at one eighth of the base η. Its optimal base η stayed around 0.01 from 0.04B to 0.5B parameters and held at 2.1B. Meta's MetaP for Llama 4 sets per-layer learning rates and initialization scales that transfer across batch size, width, depth and training tokens; the blog does not say how.
• Laws fitted on a ladder, a family of small models built by one rule, then extrapolated to the target.

MAI's rungs are set by the layer count L alone (width d = (256/3)·L, so L78 gives 6,656) and trained at a fixed number of tokens per active parameter (TPP). For MAI-Base-2, Microsoft found the best pair (η*, λ*) at several sizes and TPPs and fit two laws in the number of optimizer steps:

• The product law fixes how long the weights remember. Each step multiplies the weights by (1 − ηλ), so an old weight value fades with a time constant of about 1/(ηλ) steps. The −1 exponent makes this time constant 1/(η*λ*), which Microsoft calls the decoupled weight-decay time scale, a fixed share of training: steps/17.04, about 6%. A 100,000-step run gets η*λ* = 17.04/100,000 ≈ 1.7×10⁻⁴, so its weights forget over about 5,900 steps.
• The learning-rate law nearly reproduces a rule of thumb. The −0.573 exponent on steps is close to the rule that doubling the batch should multiply η by √2: at a fixed token budget, doubling the batch halves the steps, so η* rises by 2^0.573 ≈ 1.49, near √2 ≈ 1.41.

These laws set MAI-Base-2's hyperparameters, and Kimi K3 likewise retuned batch size, η, TPP and model shape with scaling-law studies.

Small rungs must also see data the way the big run does. MAI's source scaling shrinks the unique data on small rungs so that a dataset repeated 8 times over 50T tokens is looped 8 times over 1/10 of the data in a 5T-token run:

"Without source scaling, experiments at smaller scale can favor under-regularized architecture and hyperparameter choices that overfit at larger scales."

Inkling instead ties weight decay to η:

"We coupled the weight decay strength to the square of the learning rate, which we found kept the overall size of the model weights stable across training horizons"

1. Hu et al., MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies (2404.06395)
2. Hoffmann et al., Training Compute-Optimal Large Language Models (Chinchilla, 2203.15556)
3. Vaswani et al., Attention Is All You Need (1706.03762)
4. Xiong et al., On Layer Normalization in the Transformer Architecture (2002.04745)
5. DeepSeek-V3 Technical Report, §4.2-4.3
6. DeepSeek-V4.1-Flash Technical Report, §4.2.2
7. Liu et al., Muon is Scalable for LLM Training (Moonlight), §3.3
8. Kimi K2: Open Agentic Intelligence, §2.5
9. Kimi K3 Technical Report, §3.2-3.3
10. GLM-4.5 Technical Report, §2.4
11. MiMo-7B Technical Report, §2.3
12. MAI-Thinking-1 / MAI-Base-1 report, §2.2.1 and §2.6.2
13. Frontier Pre-training at Microsoft AI (MAI-Base-2), PDF and blog version
14. Meta, The Llama 4 herd (blog), §Pre-training
15. Thinking Machines, Introducing Inkling

10 Training stability

What makes a long pretraining run's loss spike or diverge, and how did Kimi K2 train 15.5T tokens with Muon "without a single loss spike"?

What instability looks like

A loss spike is a sudden jump in training loss, usually together with a jump in gradient norm. Some spikes recover on their own; others keep rising until training diverges and never comes back. Either way they waste compute: the steps inside the spike teach the model little, and the usual repair is to roll back to an earlier checkpoint and redo work.

Where instabilities start

A spike begins when some quantity grows past what the model or the number format can absorb. The reports point to five places where that growth starts, each with its own guards:

The sections below follow the rows in order; published incidents come after them.

Attention logits: why they grow

Softmax turns attention scores into weights. With four keys and logits (2, 0, 0, 0), the top key gets weight e² / (e² + 3) = 0.71. At (20, 0, 0, 0) it gets all but 6 × 10⁻⁹. When one logit is far above the rest, softmax becomes one-hot and stops passing gradient to the other keys.

The attention logit between query position t and key position s in head h is q_t·k_s/√d_h. Kimi bounds it by the sizes of the inputs and of the two projection matrices:

Here x is a token's input to the layer, and a matrix's spectral norm ‖W‖₂, its largest singular value, is the most it can stretch a vector. RMSNorm keeps the x terms bounded, so growth comes from the spectral norms of W_q and W_k, and since the logit uses their product, any growth is compounded.

Muon makes the growth worse; this is the optimizer row of the table. Its updates have equal singular values (full rank), while Adam's are dominated by a few directions, and Kimi's hypothesis is that full-rank updates are more likely to line up with a weight's own singular directions and push its largest singular values up. Moonlight already showed max logits above 100 in some layers early in training. In a 9B-active, 53B-total MoE trained with vanilla Muon, max logits "quickly exceed a magnitude of 1000", and K2 adds that "Max logits at this level usually result in instability during training, including significant loss spikes and occasional divergence."

QK-Clip

QK-Clip is Kimi's answer. It watches each head's largest logit and, when that passes a threshold τ, shrinks the head's query and key weights just enough to bring it back. In each training step:

1. the forward pass records each head's largest logit over the batch, S^h_max;
2. the optimizer takes its usual Muon step;
3. every head with S^h_max above τ has its query and key weights scaled down so that its logits shrink by γ_h = τ / S^h_max.

With α = ½ each matrix shrinks by √γ_h. The forward and backward pass of the current step are not changed. Clipping is per head because "only a small subset of heads exhibit exploding logits".

Why not a standard remedy? QK-norm (Chapter 5) needs explicit keys, and the keys of MLA, K2's attention, "are not fully materialized during inference". A logit soft-cap bounds the logit after the dot product, but "the dot products between queries and keys can still grow excessively before capping is applied." QK-Clip acts on the weights instead. MLA needs one adjustment, because each head's logit is a content part q_C·k_C plus a rotary part q_R·k_R: K2 scales the head-specific content parts q_C and k_C by √γ, scales the head-specific rotary query q_R by γ, and leaves the rotary key k_R alone because every head shares it.

def qk_clip(attn_layers, tau=100.0):
    # Runs right after the Muon update. S_max[h] was recorded in this
    # step's forward pass: max over the batch of q_t . k_s / sqrt(d_h).
    for layer in attn_layers:
        for h in range(layer.num_heads):
            s = layer.S_max[h]
            if s <= tau:
                continue                       # most heads, most steps
            gamma = tau / s                    # logits of head h shrink by gamma
            if layer.is_mla:
                layer.W_qc[h] *= sqrt(gamma)   # head-specific content query
                layer.W_kc[h] *= sqrt(gamma)   # head-specific content key
                layer.W_qr[h] *= gamma         # head-specific rotary query
                # W_kr (rotary key) is shared by all heads: left unchanged
            else:
                layer.W_q[h] *= sqrt(gamma)
                layer.W_k[h] *= sqrt(gamma)

K2 ran with τ = 100. Max logits rose to the cap early and stayed there; 12.7% of heads triggered QK-Clip at least once in the first 70,000 steps, after which every head stayed below 100 and the clip stopped firing. K2 pretrained on 15.5T tokens without a loss spike. In a small ablation with an aggressive τ = 30, the loss curve was almost unchanged. Kimi K3 keeps "the weight-clipping mechanism introduced in Kimi K2". Kimi calls the whole optimizer, Muon with weight decay and update-RMS matching (Chapter 08) plus QK-Clip, MuonClip. Figure 10.1 contrasts the two runs.

Logit soft-capping

Soft-capping bounds a logit smoothly: logit ← cap · tanh(logit / cap). Values well below the cap pass almost unchanged and large values approach ±cap. With cap 50, logits of 10, 100 and 1,000 become 9.87, 48.2 and 50.0. Gemma 2 used cap 50 on attention logits and 30 on the final output logits. Gemma 3 replaced Gemma 2's soft-capping with QK-norm, while Meta's Muse Glimmer sets final_logit_softcapping = 20.0 in its config. The cost is gradient: the derivative of a soft-cap is 1 − tanh²(logit / cap), which falls toward zero near the cap (0.07 at a logit of 100 under cap 50), so a capped value receives almost no gradient.

z-loss: output and router logits

Softmax depends only on differences between logits, so a whole logit vector can drift upward without changing the loss, and nothing in the loss pulls it back. Large values are a precision hazard: ST-MoE notes that bfloat16 roundoff can be up to 65,536× worse than float32, and roundoff grows with magnitude. PaLM adds an auxiliary z-loss of 10⁻⁴ · log²Z, where Z = ∑_j exp(logit_j) is the output softmax normalizer, to keep log Z near 0. ST-MoE applied the idea to the router of an MoE layer, whose logits x_ij score expert j for token i in a batch of B tokens and E experts:

It joins the training loss with a small weight:

ST-MoE compared it with the blunter alternative of tightening the update clipping in Adafactor, the memory-saving optimizer it trained with. Quality is negative log perplexity on English, so higher is better:

Outlier activations in MoE layers

Activations can blow up too, and the MoE reports describe two kinds of guard:

• Bound the activation. DeepSeek-V4's SwiGLU clamping clamps the linear part to [−10, 10] and caps the gate at 10. Kimi K3 applies the soft-cap's tanh shape inside its MoE activation (SiTU-GLU) "to suppress activation explosion".
• Break the loop through routing. DeepSeek's reasoning is that routing itself amplifies the outliers, a "vicious cycle". Anticipatory Routing computes the routing from earlier parameters θ_t−Δt, which decouples the routing decision from the weights being updated and so breaks that loop. It is switched on automatically, with a short rollback, when a spike appears. DeepSeek adds that the mechanism is not yet fully understood.

Gradient clipping and activation-gradient clipping

Global gradient clipping rescales the whole gradient when its norm exceeds a threshold c: g ← g · min(1, c / ‖g‖). DeepSeek-V3, MiMo-7B, MiMo-V2-Flash, MAI-Thinking-1 and PaLM all use c = 1.0. Its weakness is that one bad example shrinks the gradient of the whole batch. If the well-behaved examples give a gradient of norm 0.5 and one outlier adds norm 50, clipping at c = 1 divides everything by about 50 and the useful part shrinks to 0.01. Two narrower guards limit the outlier instead:

• Activation-gradient clipping. MAI-Base-2 also clips the gradients of activations at selected places in the network, which limits the backward signal of individual tokens. Together with Muon, this kept MAI-Base-2's gradient norm in a much tighter range than MAI-Base-1's, and the benefit grew with scale, to a +28% efficiency gain on held-out evaluations at the largest size of their scaling ladder. The clipped tokens double as a data probe: one led MAI to a preprocessing bug that had cut a license header mid-sentence.
• Per-layer clamps. StepFun's Step 5 Preview is narrower still: against gradient spikes it clamps only the largest 0.00004% of gradient values in each layer.

Spikes in published runs

Read against the table, most published incidents have a clear origin; some do not.

• PaLM 540B: data and parameter state together. Google saw spikes "roughly 20 times during training, despite the fact that gradient clipping was enabled", at irregular intervals and sometimes late in the run. The fix was to restart from a checkpoint about 100 steps before the spike and skip 200 to 500 batches. Each rollback repeats at least the 100 steps before the spike; at PaLM's batch sizes of 1M to 4M tokens, that is 0.1B to 0.4B tokens per spike, or 2B to 8B over 20 spikes, up to about 1% of the 780B-token run. Replaying the same batches from an earlier checkpoint did not spike, so the authors concluded that spikes come from "the combination of specific data batches with a particular model parameter state."
• MAI-Base-1: routing imbalance. Several early spikes recovered without intervention; they mostly hit coding data and correlated with "high expert imbalance under dropless routing" (dropless: no per-expert capacity limit, so every routed token is processed even when a few experts receive most of them).
• DeepSeek-V4: activations. Rollbacks "do not prevent the recurrence of loss spikes"; DeepSeek tied the spikes to outliers in the MoE layers and added the two activation guards above.
• Nemotron 3 Ultra: numerics, then unknown. NVIDIA's run diverged twice: at about 8T tokens because gradient accumulation for the output layer had dropped from FP32 to BF16, fixed by going back to FP32, and near 16T for a reason NVIDIA could not determine, after which it annealed early from the 15T checkpoint and cut the run to 20T tokens.

Removing the cause

The newest designs try to keep logits bounded without a clip. DeepSeek V4 applies RMSNorm to its queries and compressed KV entries and so does "not employ the QK-Clip technique". GLM-5's per-head Muon Split keeps "the scale of attention logits" stable "without any clipping strategy". Gemma 3 and 4, GLM-4.5 and MAI-Thinking-1 use QK-norm. Qwen relies on gates: an attention output gate in Qwen3-Next removed attention sinks and massive activations, and Qwen3.8-Flash-Next trained without a single loss spike and without QK-Clip or SwiGLU clipping, which Qwen credits largely to the gate in its Gated Residual (chapter 05).

Who uses it and why

"scaling up Muon training reveals a challenge: training instability due to exploding attention logits, an issue that occurs more frequently with Muon but less with AdamW in our experiments."

"An outlier can therefore cause the gradients from the entire batch to be scaled down, including contributions from otherwise well-behaved examples."

"Empirically, we identified that the occurrence of spikes is consistently tied to outliers in the MoE layers, and the routing mechanism itself appears to exacerbate the emergence of these outliers."

1. Kimi Team. Kimi K2: Open Agentic Intelligence, §2.1, App. D, App. E (2025).
2. Liu et al. (Moonshot AI). Muon is Scalable for LLM Training (Moonlight), App. D (2025).
3. Chowdhery et al. PaLM: Scaling Language Modeling with Pathways, §5, §5.1 (2022).
4. Zoph et al. ST-MoE: Designing Stable and Transferable Sparse Expert Models, §3 (2022).
5. Gemma Team. Gemma 2 report, §2; Gemma 3 report, §2.
6. Microsoft AI. Frontier Pre-training at Microsoft AI (MAI-Base-2); MAI-Thinking-1 report, §2.6.
7. DeepSeek-AI. DeepSeek-V4 report, §2.4, §4.2.3; DeepSeek-V3 report, §4.2.
8. Kimi Team. Kimi K3 technical report, §2.3, §3.3. Z.ai. GLM-5 report, §2.1.
9. Meta. Muse Glimmer config.json. Xiaomi. MiMo-7B report; MiMo-V2-Flash report.
10. NVIDIA. Nemotron 3 Ultra report. StepFun. Inside Step 5 Preview. Qwen. Qwen3-Next blog; Qwen3.8-Next architecture report.

11 Low-precision training

How do labs run the matrix multiplies of pretraining on 8-bit numbers without the loss drifting away from a 16-bit run?

Why the number of bits matters

Fewer bits per number mean less memory, less bandwidth, and more multiply-adds per second from tensor cores, the GPU's matrix-multiply units. DeepSeek says that running its GEMMs (general matrix multiplications) in 8-bit floating point, FP8, theoretically doubles the computational speed of 16-bit BF16. Memory adds up: Kimi K2's BF16 parameters plus an FP32 gradient buffer take about 6 TB, which is 1.04 trillion parameters times 2 + 4 bytes.

What a floating-point number stores

A floating-point number has a sign bit, exponent bits that set its range, and mantissa bits that set its precision between two powers of two: with m mantissa bits, each such span holds 2^m evenly spaced values. BF16 has 7, so it stores 0.1 as 0.10009765625.

FP16 (1, 5 and 10 bits) has a narrow range: its normal values run from 6.10×10⁻⁵ to 6.55×10⁴, so FP16 training needs loss scaling, a multiplier on the loss that keeps small gradients from underflowing. BF16 keeps FP32's 8 exponent bits with a 7-bit mantissa, so it covers FP32's range without loss scaling. Figure 11.1 draws the layouts to scale.

The standard recipe feeds BF16 tensors to each GEMM, accumulates in FP32, and keeps an FP32 master copy of the weights for the optimizer.

Two FP8 formats

The FP8 formats paper (NVIDIA, Arm, Intel) splits 8 bits two ways: E4M3 (4 exponent bits, 3 mantissa bits) and E5M2 (5 and 2). E5M2 follows IEEE conventions and reaches 57,344. E4M3 drops infinities and keeps one NaN pattern to gain a power of two: it spans 2⁻⁹ to 448. The paper recommends E4M3 for weights and activations, E5M2 for gradients. A stored value decodes as

With 3 mantissa bits, E4M3 holds only 1, 1.125, 1.25, …, 1.875 between 1 and 2, so rounding costs at most 1/16 (6.25%) of a value. Below the smallest normal value, 2⁻⁶, numbers are subnormal: they sit on a fixed grid of spacing 2⁻⁹, so relative error grows as values shrink.

In simulated FP8 (inputs rounded, arithmetic in 16-bit), the paper matched 16-bit GPT training up to 175B parameters, perplexity 6.68 against 6.65. It relied on a scaling factor, chosen so that a tensor's largest magnitude lands near the format's maximum, then divided out after the GEMM. How finely that scale is chosen is what separates the FP8 recipes below.

One scale per tensor, or one per block

The simplest choice is one scale per tensor, s = 448 / max|x|, often with delayed scaling, which infers the maximum from earlier steps. Here s multiplies values before rounding and is divided out after the GEMM. The weakness is outliers, which DeepSeek says often limit low-precision training. One huge value sets max|x| and shrinks the scale for everything else. Any value more than 448 / 2⁻⁶ = 28,672 times below the maximum falls among the subnormals or rounds to zero.

The cure is one scale per small block, so an outlier shrinks only its own block's scale. Block formats differ in block size and in how the scale is stored:

• FP8 with FP32 block scales. DeepSeek-V3 scales activations per 1×128 tile (one token, 128 channels) and weights per 128×128 block, with E4M3 values and FP32 scales computed online from the current values.
• MXFP8 and MXFP4. The Microscaling (MX) formats, an Open Compute Project standard, build the block into the format. An MX block holds 32 elements and one shared 8-bit E8M0 scale, an exponent-only number and so a power of two. MXFP8 has E4M3 or E5M2 elements; MXFP4 has 4-bit E2M1, which can hold only 0, 0.5, 1, 1.5, 2, 3, 4 and 6 and their negatives.
• NVFP4. NVFP4 stores E2M1 values with an FP8 E4M3 scale per 16 elements and one FP32 scale per tensor.

DeepSeek's FP32 scale is 448 / max|x| over each block. MX picks a power of two instead and divides by it:

The shared scale s_blk divides the block's values before rounding. e_max is the exponent of the element format's largest normal value: 8 for E4M3, since 448 = 1.75 · 2⁸, and 2 for E2M1, since 6 = 1.5 · 2². The scale adds 8 / 32 = 0.25 bits per element, which is why gpt-oss counts its MXFP4 weights as 4.25 bits per parameter. A power of two is cheap to apply but coarser than an FP32 scale. Quantizing and transposing do not commute: the backward pass multiplies by W^⊤, whose 32-element blocks run along the other dimension, so training keeps separately quantized copies of a weight and its transpose.

Block scales inside a matrix multiply

A matrix multiply sums over the inner dimension K: Y[m, n] = ∑_k x_mkw_kn. One scale per row of X and per column of W is constant along that sum and can be divided out afterwards. DeepSeek's scales change every 128 channels along K, cannot be factored out, and so cannot be applied by standard FP8 GEMMs. DeepSeek applies them during accumulation instead, slice by slice.

The hardware gives a second reason to split the sum. H800 tensor cores keep only about 14 bits in FP8 sums, which gave a maximum relative error of nearly 2% at K = 4096. So every N_C = 128 elements, partial sums move to FP32 registers on the CUDA cores (the general-purpose units), where they are unscaled and summed. Figure 11.2 draws one such GEMM.

def quantize_x(X):                        # X: [M, K] activations
    T = X.reshape(M, K // 128, 128)       # 1x128 tiles
    s = 448.0 / T.abs().amax(-1)          # [M, K/128], one scale per tile
    return to_e4m3(T * s[..., None]).reshape(M, K), s

def quantize_w(W):                        # W: [K, N] weights
    B = W.reshape(K // 128, 128, N // 128, 128)    # 128x128 blocks
    s = 448.0 / B.abs().amax((1, 3))      # [K/128, N/128]
    return to_e4m3(B * s[:, None, :, None]).reshape(K, N), s

def fp8_gemm(Xq, sx, Wq, sw):             # returns Y: [M, N] in FP32
    Y = zeros(M, N, dtype=fp32)
    for k in range(K // 128):             # one 128-wide slice of K
        cols = slice(128 * k, 128 * (k + 1))
        P = Xq[:, cols] @ Wq[cols, :]     # tensor cores, ~14-bit sums
        sw_k = sw[k].repeat_interleave(128)          # [N]
        Y += P / (sx[:, k, None] * sw_k[None, :])    # FP32 on CUDA cores
    return Y

Small groups let DeepSeek use E4M3 even for gradients, as grouping effectively shares exponent bits: each group's scale acts as a shared exponent, so an element's own exponent only covers the range inside its group, where E4M3's 4 bits suffice. Embedding, output head, MoE gating, normalization and attention stay in BF16 or FP32. The tile shape matters: with 128×128 blocks for activation gradients, a 16B MoE diverged after about 300B tokens, which DeepSeek hypothesizes comes from token-correlated outliers. Against BF16 runs at about 16B and 230B parameters, the final recipe stayed below 0.25% relative loss error.

Who trains in which format

The published pretraining recipes sort into five groups:

• FP8 with one scale per tensor. MAI-Thinking-1 used E4M3 forward GEMMs, E5M2 for the GEMM that computes gradients with respect to activations, and delayed scaling with a 1,024-step history of the absolute maximum. NVIDIA used one scale per tensor in Nemotron-H, with the first and last four layers in BF16.
• FP8 with DeepSeek's blocks. DeepSeek presents V3 as the first validation of FP8 training on an extremely large model. NVIDIA then moved to DeepSeek's blocks for Nemotron Nano 2's whole run; Ant trained every Ling 2.0 model, up to 1T parameters, entirely in blockwise FP8.
• MXFP8. MAI-Base-2 replaced MAI-Base-1's tensor-wise delayed E4M3/E5M2 scheme with MXFP8 E4M3; its report gives no reason. DeepSeek called its tiles highly consistent with the idea of microscaling formats, and DeepSeek-V3.1 moved to the UE8M0 scale format for MX compatibility.
• FP8, other recipes. MiMo-V2-Flash uses a framework similar to DeepSeek-V3. Llama 4 trained in FP8, reaching 390 TFLOPs per GPU for Behemoth on 32K GPUs. Qwen3.5 runs activations, MoE routing and matrix multiplies in FP8, with runtime monitoring that keeps sensitive layers in BF16, and StepFun's Step 5 Preview trains in FP8 while keeping its hidden states, their gradients and its RMSNorm scales in FP32.
• BF16 compute, FP8 storage. Kimi K2 computes with BF16 parameters and FP32 gradient accumulation and uses FP8 only for storage: inputs to the MoE up-projections and SwiGLU are saved for the backward pass as E4M3 in 1×128 tiles with FP32 scales.

GLM-5, Gemma 4, Inkling, Qwen3, MiniMax and Mistral do not state their pretraining precision.

FP4 is mostly a post-training format

Four-bit formats enter mainly through quantization-aware training (QAT): the forward pass sees quantized weights while gradients update a high-precision copy. gpt-oss quantized its MoE weights to MXFP4 in post-training. DeepSeek-V4's MXFP4 expert weights dequantize losslessly to FP8 provided the largest and smallest 1×32 FP4 scales inside each 128×128 FP8 block differ by less than a set ratio, which DeepSeek checked empirically for its weights; its FP8 framework then runs unchanged. Kimi K3 runs MXFP4 expert-weight, MXFP8-activation QAT through all of post-training, and MiMo-V2.6 starts MXFP4 QAT in mid-training. The posttraining notes compares these schemes.

NVIDIA is the exception. It pretrained Nemotron 3 Super in NVFP4 for most linear layers over all 25T tokens, keeping the last 15% of the network, the attention projections and the embeddings in BF16; Nemotron 3 Ultra followed over 20T tokens. Underflow was the visible cost: by the end of Super's run, zero-valued weight gradients covered 7% of the parameters.

Who uses it and why

"Through the support for FP8 computation and storage, we achieve both accelerated training and reduced GPU memory usage."

"Due to potential risks of performance degradation that we observed during preliminary study, we do not apply FP8 in computation."

1. Micikevicius et al., FP8 Formats for Deep Learning (2022), arXiv:2209.05433
2. Rouhani et al., Microscaling Data Formats for Deep Learning (2023), arXiv:2310.10537
3. Kalamkar et al., A Study of BFLOAT16 for Deep Learning Training (2019), arXiv:1905.12322
4. DeepSeek-AI, DeepSeek-V3 Technical Report, §3.3 and App. B
5. DeepSeek-V3.1 model card
6. DeepSeek-V4 technical report, §5.2.1
7. DeepSeek-V4.1-Flash technical report, §2.4.4
8. Kimi Team, Kimi K2 technical report, §2.4
9. Kimi Team, Kimi K3 technical report, §4.1.4 and §5.2.2
10. Microsoft AI, MAI-Thinking-1 report, §2.6.3
11. Microsoft AI, Frontier Pre-training at Microsoft AI (MAI-Base-2)
12. Meta, The Llama 4 herd (launch blog)
13. Xiaomi, MiMo-V2-Flash technical report, §2.1
14. Xiaomi, MiMo-V2.6 technical report, §3.2
15. OpenAI, gpt-oss-120b and gpt-oss-20b model card, §2.1
16. NVIDIA, Nemotron-H report, Nemotron Nano 2 report, Nemotron 3 Super report, Nemotron 3 Ultra report and Nemotron 3 Nano Omni report
17. Ant, Ling 2.0 report; Qwen, Qwen3.5 blog; StepFun, Inside Step 5 Preview

12 Data

Where do the 14 to 50 trillion training tokens of a large 2025-26 base model come from, and how do labs choose, clean, rewrite and check them?

How many tokens

How many tokens should a model of a given size see? A pretraining run reads a budget of D tokens. The Chinchilla study trained over 400 models and found that, for compute-optimal training, "the model size and the number of training tokens should be scaled equally". Chinchilla itself had 70B parameters and saw 1.4T tokens: 1.4×10¹² / 7×10¹⁰ = 20 tokens per parameter. Compute-optimal counts training cost only; a smaller model also "reduces inference cost considerably".

Open models train far past that ratio:

• Many tokens per parameter. Gemma 3 27B saw 14T tokens, about 520 per parameter or 26 times Chinchilla's ratio, and the dense MiMo-7B saw 25T, about 3,200 per parameter (its checkpoint holds 7.8B weights).
• Growing budgets. 14.8T for DeepSeek-V3 (December 2024) and 15.5T for Kimi K2 (July 2025), then 27-50T in 2026, up to 50T for MAI-Base-2 (Figure 12.1).
• Labs added in the atlas. Qwen3 was already at 36T in April 2025, MiniMax M2 used 29.2T in October 2025, and NVIDIA's Nemotron-H-56B and Ant's Ling 2.0 each used 20T; Qwen3.5 and 3.8, MiniMax M3, Step 5 and Mistral's models give no token count.

Labs count differently (MiMo includes later mixture stages), so small gaps mean little.

What goes into the mix

Given a budget, how much of it should each kind of text get? A data mix gives each source family i (web, code, math, books) a share of the run, and the share decides how often that family's text is repeated:

U_i is what cleaning left of the family, so e_i is how many times it is seen on average; e_i above 1 means repetition. A 10T-token run that gives code w = 20% draws 2T code tokens; with 1T unique code tokens, each is seen e = 2 times.

Most labs describe the mix only in words. DeepSeek-V3 raised "the ratio of mathematical and programming samples", and Kimi K2 names four domains without shares. A few give numbers:

• MiMo-7B raised math and code to about 70% in its second stage.
• GLM-4.5 trained on 15T general tokens, then 7T of code and reasoning.
• Ant raised Ling 2.0's reasoning share from 32% to 46% over its 20T tokens.
• DeepSeek-V4.1-Flash mixes text-only and multimodal tokens 7:1.
• NVIDIA publishes phase-by-phase mixes for its Nemotron models, and Microsoft AI (MAI) publishes full mixes for MAI-Base-1, the base of MAI-Thinking-1, and for MAI-Base-2 (Figure 12.2 and the worked example below).