00 What midtraining is and why it grew

What happens between the main pretraining run and fine-tuning, and why did it grow from a 126B-token context patch in DeepSeek-V3 (0.8% of its tokens) to 17T in DeepSeek V4.1-Flash (38%) and about half of Kimi K2.5's training?

A language model is built in three stages. Pretraining starts from random weights and trains next-token prediction on a large general corpus of web text, code, books and papers. Posttraining comes last: supervised fine-tuning (SFT, training on example responses) and reinforcement learning (RL, training on rewards for the model's own outputs) teach the model how to answer, reason and use tools. Midtraining sits between them. The loss is still next-token prediction on documents, but the recipe changes: smaller, targeted data, longer sequences, a low learning rate η (the factor that scales each weight update), and sometimes new modules such as a vision encoder.

The report on GLM-4.5, Z.ai's 355B-parameter mixture-of-experts (MoE) model, states the idea plainly: after pre-training it adds stages that use "medium-size domain-specific datasets, including instruction data", and it names them mid-training.

Where the boundaries sit

Labs use different words for these phases: mid-training (GLM, MAI, Meta, MiMo, DeepSeek V4), annealing (Kimi K2), cooldown (Moonlight, Kimi K3), learning-rate decay stage (DeepSeek V4.1-Flash), context extension (DeepSeek-V3) and continued pre-training (DeepSeek V3.2-Exp). To compare releases, these notes apply one rule to every lab. The idea behind it: midtraining is a deliberate change of recipe after the main run, not just the tail of a learning-rate schedule. A phase shows that intent in one of three ways: the lab names it as a phase, it changes the data or the context after the main decay, or it continues training a model that was already trained. In full:

• Pretraining is the from-scratch run on its main learning-rate schedule, including any decay inside that schedule, data re-weighting during it, and sequence-length growth the lab does not name as a separate phase.
• Midtraining is (1) any phase a lab names mid-training, annealing, cooldown, learning-rate decay stage, context extension or continued pre-training; (2) a distinct stage after the main decay that changes the data mix or the context length (MiMo-7B stages 2-3); (3) continued training of an earlier base, such as adding vision, a new attention or longer context. The earlier base can be a released model or an intermediate checkpoint of an earlier run.
• Posttraining starts at SFT.

Figure 0.1 lays the rule out with an example under each entry.

The rule sometimes overrides the lab's own label. Four cases show how it sorts real phases:

• Kimi K2, Moonshot AI's 1.04T-parameter MoE, decays η over its last 5.5T pretraining tokens. Kimi calls that pre-training and so does the rule, since the decay sits inside the main schedule; only the following 400B annealing and 60B long-context phases count.
• MiMo-7B, Xiaomi's 7B model, calls all three of its stages pre-training, but stages 2 and 3 come after the main decay and raise math and code to about 70% of the mix, so clause (2) counts them as midtraining.
• Kimi K2.5 and MiMo V2.6. Their labs call K2.5's vision-text stage and V2.6's omni stage pre-training. Clause (3) counts both as midtraining because they continue training an already pretrained text model (K2.5 starts from a near-end K2 checkpoint).
• Step 3.5 Flash (Feb 2026). StepFun files a 3T-token annealing stage under pre-training and calls only the 750B tokens after it mid-training. Because StepFun names the stage annealing, clause (1) counts it, which raises the release's midtraining share from about 4% to about 20%.

Why not count every learning-rate decay? A literal "any decay" rule would add about 4.8T to DeepSeek-V3 and 5.5T to Kimi K2, but nothing to labs that run one cosine from the first step (GLM, MAI). The count would then measure the shape of a schedule rather than a deliberate change of recipe.

MID_NAMES = {"mid-training", "annealing", "cooldown", "learning-rate decay stage",
             "context extension", "continued pre-training"}

def stage(phase, main_decay_end):
    if phase.kind in ("SFT", "RL"):
        return "posttrain"
    if phase.lab_name in MID_NAMES:          # clause (1)
        return "midtrain"
    if phase.start >= main_decay_end and (phase.new_mix or phase.new_context):
        return "midtrain"                    # clause (2), e.g. MiMo-7B stages 2 and 3
    if phase.continues_earlier_base:         # clause (3): released model or earlier checkpoint
        return "midtrain"
    return "pretrain"                        # incl. in-schedule decay, unnamed length growth

def midtrain_share(phases, main_decay_end):
    lm = [p for p in phases if not p.encoder_only]          # tokens must pass through the LM
    pre = sum(p.tokens for p in lm if stage(p, main_decay_end) == "pretrain")
    own = sum(p.tokens for p in lm
              if stage(p, main_decay_end) == "midtrain" and not p.inherited)
    return own / (pre + own)   # inherited pretraining counts in pre; inherited midtraining does not

Seven kinds of phase

What do labs do in these phases? Labeling each phase by its main purpose gives seven kinds. A phase that does two things is counted once, under its main purpose: GLM-4.5's 128K stage with synthetic agent trajectories is a long-context phase. The counts come from a cross-lab grid that codes 68 releases from the labs these notes started with (the "coded releases"). Of them, 27 describe a midtraining recipe of their own; the rest are refreshes that reuse a parent's phases, releases that describe none, and a few where the question does not apply (the table in the disclosure section below lists them). Among the 27, the number of releases with each kind:

• Long context (21): train on longer sequences, often with a changed position encoding; DeepSeek-V3's 4K to 32K to 128K phases (ch04, ch05).
• Multimodal (9): attach a vision or audio encoder to a trained language model and train on mixed data, as Kimi K2.5 did (ch07).
• Anneal (7): lower η, usually on upweighted high-quality data, like the cooldown of Moonlight, Moonshot's 16B MoE trained with the Muon optimizer, over 500B tokens (ch01).
• Agentic (3): tool-use and agent trajectories, in DeepSeek V4, MiMo V2.6 and Muse Glimmer (ch03).
• Code (2): code-centered stages, GLM-4.5's 500B repository-level stage (ch03) and MiMo V2-Flash's code-upsampled stage (ch02).
• Reasoning (2): synthetic reasoning traces, in GLM-4.5 and Muse Glimmer (ch02).
• Retrofit (2): change the architecture and keep training; DeepSeek V3.2-Exp and GLM-5 turned dense attention into sparse attention (ch06).

The six labs the atlas added later are not in these counts. Their reports mostly add cases to the same kinds, among them more retrofits (Qwen's sparse attention and Ling's hybrid linear attention, ch06) and more agentic stages (StepFun, NVIDIA, ch03). One kind is new: training an existing base on a larger model's logits, as NVIDIA does for both Nemotron 3.5 models; Mistral distills its Ministral 3 models from Mistral Small 3.1 the same way after pruning them from it (ch06).

How large it became

To compare releases of different sizes, these notes measure midtraining as a share of the tokens a release was trained on:

A release that pretrains on 9T tokens and then midtrains on 1T has a share of 1 / (9 + 1) = 10%. Two details matter for real releases. D_pre includes pretraining inherited from a parent model, while D_mid counts only the release's own midtraining tokens. And only tokens that pass through the language model count, so a vision encoder trained on its own is left out. Figure 0.2 applies the measure to nine releases, in release order and to scale.

In tokens, the growth runs from billions to trillions:

• 2024. DeepSeek-V3, DeepSeek's 671B MoE (37B parameters active per token), spent about 126B tokens on two context-extension phases.
• 2025. DeepSeek-V3.1 re-ran both phases from the V3 base with 630B and 209B tokens. Outside the grid, MiniMax was already spending trillions: 7.5T continuing MiniMax-Text-01 into M1 (Jun 2025), and 9.3T, 32% of M2's tokens, in M2's named decay phase (Oct 2025).
• 2026. Several releases spent trillions. MAI-Thinking-1, Microsoft AI's roughly 1T-total MoE (962B in its parameter table), spent 3.55T; MiMo V2.6-Pro, Xiaomi's 1.02T MoE, at least 3T (its agent-centric phase is not sized); DeepSeek V4.1-Flash, a 552B MoE, 17T in its named learning-rate decay stage and 1M-token context extension; and Kimi K2.5's vision-text continued pretraining 15.7T, more than the roughly 15T of K2 text pretraining it branches from (K2 itself ran 15.5T).

Part of this rise is labeling. The rule follows what a lab names as a phase, and three DeepSeek generations land on different sides of that line:

• DeepSeek-V3 decays η inside its main schedule without naming a stage, so the rule keeps that decay in pretraining. The worked example below shows how much that matters, and ch10 returns to it.
• DeepSeek V4 (April 2026) grows its sequence length from 4K to 1M inside its single run, which the rule counts as pretraining, and names an agentic mid-training phase of unstated size (ch03).
• DeepSeek V4.1-Flash (September 2026) is a new model trained from scratch at 64K, not a continuation of DeepSeek V4 (the roster maps the family). It names its learning-rate decay stage and its context extension, which these notes place at 28T to 34T and 34T to 45T (η keeps falling until 40T and is then held, ch01), so all 17T count. Besides a falling η, those 17T change two things: the context grows from 64K to 1M at 34T, and the vision encoder, frozen until the decay starts at 28T, trains from then on.

Growth is not monotone. MiMo-7B already put a quarter of its tokens into stages 2 and 3 in April 2025, while GLM-5, Z.ai's 744B MoE, spent 5.5% in February 2026.

Why labs make changes late

Why did so much work move into this late stretch, rather than into pretraining or a new run? The reports and papers give four reasons.

The last data have an outsized effect. When η decays, the loss falls fastest: MiniCPM saw it drop sharply as soon as the decay began. Putting data there also beats saving it for SFT: mixing high-quality and SFT data into a 2.4B model's decay raised MMLU from 44.6 to 50.9 after identical SFT. Chapter 01 shows why.

It costs a fraction of a new run. A change made late needs only the late tokens. DeepSeek gave a V3.1-Terminus checkpoint sparse attention (creating V3.2-Exp) with 945.8B tokens of continued training instead of a new 14.8T run.

Long sequences are cheapest at the end. A token in a long sequence costs more than one in a short sequence, because standard attention compares it with every earlier token: DeepSeek-V3's context extension used several times more GPU-hours per token than its pretraining (ch05 works out the numbers). Llama 3's report gives this as the reason to train on them only at the end: "the compute in self-attention layers grows quadratically in the sequence length". Kimi says of K3's context curriculum, which reaches 256K and 1M only in its cooldown, that "concentrating the costly long-sequence computation within a small fraction of the overall training budget keeps the curriculum economical".

It prepares the base for RL. MiMo V2.6 switched its optimizer from AdamW to Muown, a Muon variant, during mid-training "to prepare the model for subsequent large-batch RL", and started quantization-aware training there, training with the forward pass in the low-precision number format the model will later run in, so it adapts to it (ch08).

Who uses it and why

"In mid-training we further emphasize STEM, math, and coding abilities to build a strong foundation for reasoning RL climbs."

"We further introduce an agent-centric mid-training phase that expands the exploration space for agentic tasks, enabling the model to discover more effective task-solving trajectories during post-training."

Meta gives similar reasons: the first phase of Muse Glimmer, Meta's 30B on-device model, adds synthetic and domain data "to improve the starting point for post-training", and Llama 4's mid-training aimed "to improve core capabilities", including long context. Some labs state what they did without saying why. Kimi K2's report says only that "towards the end of pre-training, we conducted an annealing phase", with no reason and no description of its data.

The disclosure gap

The 68 coded releases fall into four groups:

So 40 releases describe some midtraining (27 recipes plus 13 refreshes), 25 describe none and 3 have none of their own to describe. Among the 25 are recent major releases such as Gemma 4, Inkling, MAI-Base-2, gpt-oss and Muse Spark Open. All six labs added to the atlas later describe some midtraining, so 11 of the atlas's 13 labs do; Gemma and Thinking Machines are the two that do not.

What is described is often thin:

• A name only. Some of the six later labs say little: Mistral's Leanstral 1.5 names a mid-training stage and nothing more about it.
• A result without the method. Some releases that describe no midtraining still raised the context window: Kimi K2-Instruct-0905 went from 128K to 256K and GLM-4.6 from 128K to 200K.
• Data in words, not shares. Where phases are described, the data is usually described in words; MAI-Thinking-1 (35% STEM and math, 55% code) and MiMo-7B are among the few that give shares (ch02; ch09 counts how many give token counts and mix shares).

1. DeepSeek-AI. DeepSeek-V3 Technical Report (2024), §4.1-4.3 and Table 1
2. DeepSeek-V3.1 model card
3. DeepSeek-V3.2-Exp report, §2.1
4. DeepSeek-V4.1-Flash Technical Report, §4.2.2
5. Kimi Team. Kimi K2: Open Agentic Intelligence (2025), §2.5
6. Kimi Team. Kimi K2.5 technical report (2026), §4.1-4.3, Table 3
7. Kimi K3 technical report, §3.4
8. Liu et al. Muon is Scalable for LLM Training (Moonlight, 2025), §3.3
9. GLM-4.5 Team. GLM-4.5 technical report (2025), §2.2-2.4
10. GLM-5 technical report (2026), §2
11. Xiaomi LLM-Core. MiMo: Unlocking the Reasoning Potential of Language Model (2025), §2.1-2.3
12. MiMo-V2.6 technical report, §1, §3
13. Microsoft AI. MAI-Thinking-1 technical report (2026), §2.5-2.6
14. Meta. Muse Glimmer report (2026), §4.2
15. Meta. The Llama 4 herd (blog, 2025)
16. Llama Team, Meta. The Llama 3 Herd of Models (2024), §3.4.2
17. Hu et al. MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies (2024), §4.3, §5
18. Kimi-K2-Instruct-0905 model card; GLM-4.6 model card
19. Atlas, midtraining section

01 Learning-rate decay and annealing

Why do labs put their best data in the final, low-learning-rate stretch of training, and how do short anneals tell them which data deserves to be there?

The decay phase is the end of a learning-rate schedule, where the learning rate η falls from its peak η_max toward a floor η_min. Labs also call it annealing or cooldown, and often change the data at the same time. The pretraining notes cover warmup, cosine and warmup-stable-decay (WSD) schedules in more detail (pretraining notes, ch 09).

Schedule shapes

Every schedule warms up briefly to η_max and ends low; what differs is when the fall happens (Figure 1.1):

• Cosine starts falling right after warmup and follows half a cosine down to η_min, so the whole run is one long decay with no clear start to a decay phase.
• WSD holds η at its peak for most of the run, then decays it over a short final stretch: a natural slot for chosen data, and the checkpoint just before it can be kept and decayed again later.

What the decay does to the loss

Two observations seem to clash. Each optimizer step moves the weights by an amount that scales with η: MiniCPM measured the largest element-wise weight change between checkpoints of a 2.4B model and found that "the updates exhibit a robust correlation with the learning rate's magnitude". Yet the loss barely moves while the steps are large and drops fast once they shrink. MiniCPM saw large updates with little loss reduction before the decay, and an accelerated decrease during it; Hägele et al. found the drop "aligns closely with the decay in LR".

Wen et al. explain this with a river valley picture of the loss surface: a few steep directions (the valley walls) and a flat direction along the valley floor (the river). At high η the weights "undergo large oscillations" across the walls while moving quickly along the river, so the loss stays high even as the weights make progress. Decay calms the oscillation, "revealing true optimization progress".

Why the last data count most

The river valley explains why the loss drops during the decay, not which data the final weights remember. A simple picture does: each step pulls the weights part of the way toward what its batch prefers, and every later step pulls again, partly erasing the earlier pulls. A batch's say in the final weights therefore depends on two things:

• How hard it pulled, which scales with its own η;
• How much pulling came after it, which grows with the sum of η over all later steps, the learning-rate area still to come.

With easy numbers: if every step pulls 5% of the way, then 100 steps later 0.95¹⁰⁰ ≈ 0.6% of a batch's pull is left; at 0.2% per step, 0.998¹⁰⁰ ≈ 82% is left. Big pulls are forgotten quickly, small ones remembered for long.

A one-parameter toy model makes this precise. Take one weight p and a quadratic loss per batch, ℓ_t(p) = ½ h (p − m_t)², where m_t is the value batch t prefers and h is the curvature: a large h is a steep valley wall, a small h the flat floor. After N gradient steps, p is a weighted average of the batches' preferred values, and batch t gets the weight

In a steep direction (large h) only the last stretch survives, and after a decay that stretch is the decay data. In a flat direction the average spans the whole run. Figure 1.2 runs it on a WSD schedule with the same two pull sizes at peak η: the decay supplies 99.4% of the final value in the steep direction and 18% in the flat one.

def final_weights(etas, h):
    """Weight c[t] of batch t's preferred value in the final parameter (toy SGD)."""
    c, keep = [0.0] * len(etas), 1.0
    for t in reversed(range(len(etas))):
        a = etas[t] * h
        c[t] = a * keep              # its pull, shrunk by every later step
        keep *= 1 - a
    return c, keep                   # sum(c) + keep == 1; keep stays on p_0

etas = [1.0] * 800 + [1 - k / 200 for k in range(200)]       # WSD, peak 1
for h in (0.05, 0.002):
    c, keep = final_weights(etas, h)
    print(h, round(sum(c[800:]), 3))                         # 0.994, 0.182

The model is illustrative: it assumes a fixed quadratic loss and plain SGD. Adam rescales each coordinate's step, but MiniCPM trained with Adam, and its step sizes still tracked η.

Experiments show the same effect. MiniCPM mixed high-quality and SFT data into a 2.4B model's decay; after the same SFT, MMLU rose from 44.6 to 50.9 and GSM8K from 27.7 to 42.3, and the authors recommend that capability work "should start from the decay phase". OLMo 2 calls its final stage mid-training, at 5–10% of training FLOPs; for the 7B it raised the benchmark average from 53.0 to 62.9 and GSM8K from 24.1 to 67.5, though its mid-training mix includes the GSM8K training split.

Decay shapes and lengths

A decay has a shape and a length. For the shape, let x run from 0 at the start of the decay to 1 at its end; over a decay of N_d steps ending at step N:

A quarter of the way through (x = 0.25), linear decay is at 0.75 η_max and 1−sqrt already at 0.5 η_max; halfway, they are at 0.5 and 0.29. So 1−sqrt drops η quickly at first and spends longer near zero (Figure 1.1). A linear decay to zero is common in short anneals (Llama 3, OLMo 2, Moonlight's cooldown). Hägele et al. found that the 1−sqrt shape "consistently performs better than linear decay".

On length, MiniCPM found 10% of tokens enough and 2.5% short; Hägele et al. saw gains plateau at around 20%, and a 5% (1−sqrt) decay nearly matched cosine on a 20B-token run.

Annealing as a cheap test of data

Because the decay phase reacts strongly to its data, a short anneal on a candidate dataset is a cheap way to judge it. The procedure:

1. start from a partly trained checkpoint;
2. mix the candidate into the default data at a fixed share;
3. decay η to zero over a short run on that mix;
4. compare with the same anneal on the default mix alone.

Llama 3 took a 50%-trained 8B model, annealed η linearly to 0 over 40B tokens, and gave the new dataset 30% of the mix, which it found "more efficient than performing scaling law experiments for every small dataset". The gains can shrink with scale: annealing on GSM8k and MATH training data raised Llama 3 8B's validation scores by 24.0% and 6.4%, but the 405B gain was negligible.

def anneal_test(ckpt, candidate, default_mix, tokens=40e9, share=0.3):
    """Llama 3 recipe: 50%-trained 8B checkpoint, linear decay to 0 over 40B tokens."""
    mix = {candidate: share, default_mix: 1 - share}
    model, eta0 = copy(ckpt), ckpt.lr
    steps = int(tokens / ckpt.batch_tokens)
    for k in range(steps):
        model.lr = eta0 * (1 - k / steps)      # linear anneal to 0
        model.step(sample(mix))
    return evaluate(model)    # compare with the same anneal on default_mix alone

OLMo 2 made this cheaper still with microanneals: short anneals of one candidate source mixed with general web text, with η driven to zero as in a full anneal. Its default takes equal amounts of candidate and web data (50/50), and it also varied the ratio (ch02 gives that result). 19 of them cost 130B tokens, less than three full 50B anneals. MAI tuned MAI-Thinking-1's mid-training weights "according to single-source microanneals".

What the labs did

Figure 1.3 draws six published schedules to scale. They show three ways to place midtraining around the decay; a fourth, re-warming, restarts from a decayed model.

Decay inside pretraining, then midtraining at a low η. DeepSeek-V3 ended pretraining at 7.3×10⁻⁶ and ran its two context phases at that rate, "matching the final learning rate from the pre-training stage". MiMo-7B held 3×10⁻⁵ through stage 2 and most of stage 3, then decayed to 1×10⁻⁵. Kimi K2's 5.5T cosine stays in pretraining under the rule; its annealing and long-context phases (400B + 60B) took η from 2×10⁻⁵ to 7×10⁻⁶, and the report describes neither the data nor a reason.

One cosine through midtraining. GLM-4.5 decays from 2.5×10⁻⁴ to 2.5×10⁻⁵ until the end of mid-training. MAI-Thinking-1's main cosine ends at 2×10⁻⁵; its mid-training phase 1 continues from 2×10⁻⁵ to 1×10⁻⁵ without warmup, and phase 2 runs at 1×10⁻⁶.

A named decay stage. DeepSeek V4.1-Flash holds η at 2.6×10⁻⁴ to 28T tokens. Its named learning-rate decay stage covers 28T to 34T; η keeps falling during the 1M extension (34T to 45T) until it reaches 2.6×10⁻⁵ at 40T and is then held. When the decay starts at 28T, the vision encoder is unfrozen. MiniMax named such a stage much earlier: MiniMax-Text-01 (Jan 2025) closed its main run with a 1T-token "fast decay phase" that took η exponentially to 3×10⁻⁵, and MiniMax M2 (Oct 2025) spent its last 9.3T tokens in a decay phase that also grew the context from 8K to 192K.

Re-warming a decayed model. Moonlight, after a full cosine, raised η to 1×10⁻⁴ in 100 steps, then decayed linearly to 0 over 500B tokens of "the highest quality data, focusing on math, code, and reasoning". Moonlight is not in the figure, and neither is Muse Glimmer, which runs both mid-training phases below the pretraining peak and gives no values.

Checkpoint averaging

Averaging is another way to calm the oscillation: at a constant high η, nearby checkpoints scatter around the valley floor and their average lands closer to it, since "averaging reduces noise". The uniform average of the last K checkpoints, spaced Δ steps apart and ending at step N, is

With K = 4 and Δ = 600, that is steps N, N − 600, N − 1,200 and N − 1,800. Hägele et al. found weight averaging narrows the gap to a cooldown without closing it, and ByteDance found averages of constant-η checkpoints a "low-cost simulator for the annealed performance". Labs use averaging for four different jobs:

• To preview a decay. DeepSeek-V3 kept an exponential moving average of its weights "for early estimation of the model performance after learning rate decay".
• To finish an anneal. Llama 3 405B averaged checkpoints during its final anneal; OLMo 2 7B averaged three 50B-token anneals run with different data orders.
• To start midtraining. Muse Glimmer averaged its last checkpoint with three earlier ones spaced 600 steps apart (K = 4, Δ = 600, the example above) and started mid-training from that average.
• To replace the decay. Since Ling 2.0, Ant's Ling runs hold η constant until training ends, and the final "annealing" comes from merging checkpoints: Ling-1T averages the 32 best checkpoints of its final mid-training stage. Ant calls the schedule warmup-stable-merge (WSM) and kept it in Ling 2.6 and 3.0.

Who uses it and why

"decays along a cosine schedule from a peak of 2×10⁻⁴ to a minimum of 2×10⁻⁵—a final-to-peak ratio of 0.1× rather than the more common 0.01× as we found that decaying the LR less improved post-RL results."

"Our early experiments showed that models trained with the WSD schedule perform worse on general benchmarks (SimpleQA, MMLU), indicating underfitting in the stable stage."

"The learning rate we use for this phase is lower than the peak pre-training learning rate, so as to to balance gains on synthetic-data objectives against acceptable regression on organic perplexity benchmarks."

1. Hu et al. MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies (2024), §4.3-4.4, §5
2. Hägele et al. Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations (2024), §3-4
3. Wen et al. Understanding Warmup-Stable-Decay Learning Rates: A River Valley Loss Landscape Perspective (2024)
4. Llama Team, Meta. The Llama 3 Herd of Models (2024), §3.1.3, §3.4.3
5. Team OLMo. 2 OLMo 2 Furious (2024), §2, §4
6. ByteDance Seed. Model Merging in Pre-training of Large Language Models (2025)
7. DeepSeek-AI. DeepSeek-V3 Technical Report (2024), §3.2, §4.2-4.3
8. DeepSeek-V4.1-Flash Technical Report, §2.5, §4.2.2
9. Kimi Team. Kimi K2: Open Agentic Intelligence (2025), §2.5
10. Liu et al. Muon is Scalable for LLM Training (Moonlight, 2025), §3.3
11. Xiaomi LLM-Core. MiMo: Unlocking the Reasoning Potential of Language Model (2025), §2.3
12. GLM-4.5 Team. GLM-4.5 technical report (2025), §2.4
13. Microsoft AI. MAI-Thinking-1 technical report (2026), §2.5.4, §2.6.2
14. Meta. Muse Glimmer report (2026), §4.2

02 Choosing the midtraining data mix

When a lab starts a midtraining phase with a fixed token budget, which data does it raise in the mix, by how much, and how does it keep the model from losing what pretraining taught it?

A midtraining phase has a fixed token budget, so every token given to one source is taken from another. Raising the share of math or code is the point of the phase, but it pulls against two costs:

• Repetition. The best sources are small, so a larger share means more passes over the same documents. Past about four passes, extra passes add little, and sources the model has memorized need caps.
• Forgetting. The share comes out of general data, and a model that sees less general data drifts from what pretraining taught it.

The sections below work out the arithmetic of repetition, show what labs raise and by how much, and then turn to replay and to how the weights are chosen.

Mixture weights, tokens and epochs

A corpus is split into sources such as web text, code, math and books. The mixture weight w_k is the fraction of a phase's tokens drawn from source k (∑_k w_k = 1). For a phase of D_phase tokens and a source with U_k unique tokens, three numbers follow from it:

Upweighting a source means raising w_k above its pretraining value, so u_k > 1. With easy numbers: a 1T-token phase that gives math 20% takes 200B math tokens; if the math source holds 100B unique tokens, that is 2 epochs, and if pretraining gave math 5%, math is upweighted 0.20 / 0.05 = 4 times. Good sources are small, so a large w_k often means e_k > 1: repetition.

Repetition has a limit. In runs up to 9B parameters, Muennighoff et al. found that up to 4 epochs of repeated data changed loss negligibly compared with unique data, while with more repetition the value of extra compute eventually decays to zero. MAI caps every dataset at 8 repetitions in its pretraining mix search, and its ~300B tokens of high-quality math were already seen 5.28 times on average before midtraining. Xiaomi reports one exception: in its preliminary experiments, synthetic reasoning data could be trained for an extremely high number of epochs without overfitting.

What labs upweight

The simplest recipe is a cooldown: decay the learning rate (chapter 01) on the best data. Moonlight, Kimi's 16B-total, 3B-active mixture-of-experts (MoE) model, re-warmed its learning rate and decayed it to zero over a 500B-token cooldown on "the highest quality data, focusing on math, code, and reasoning", without percentages.

Xiaomi's MiMo-7B, a 7B dense reasoning model, gave numbers. Xiaomi calls its stages 2 and 3 pre-training; these notes count them as midtraining because they follow the main learning-rate decay and change the mix. Stage 2 (4T tokens) raised math and code to about 70%. Stage 3 (about 2T) added about 10% synthetic reasoning responses. MiMo-V2-Flash, Xiaomi's 309B-total, 15B-active MoE, upsampled code-centric data in its 4T-token "Mid-training" stage and added about 5% synthetic reasoning data; the code share is not given.

Microsoft AI's MAI-Thinking-1 gives the fullest numbers. Its base model, MAI-Base-1, is a 962B-total, 34.7B-active MoE. MAI-Thinking-1's first midtraining phase, 3.4T tokens at 65,536 context, sets STEM and math to 35%, keeps code at 55% and leaves 10% for background sources, all drawn from the pretraining corpus: "no new or synthetic sources are introduced". Meta's Muse Glimmer, a ~29.6B dense model built for on-device agents, adds synthetic reasoning traces and structured problem-solving demonstrations to organic data in its midtraining Phase 1, without shares.

Moonlight, GLM-4.5, Kimi K2.5, DeepSeek V4, MiMo V2.6 and Muse Glimmer say what they upweighted but not by how much. Figure 2.1 collects the phases that state percentages. Among the labs added to the atlas later, MiniMax states a share close to MiMo-7B's: M1's 7.5T-token continued pretraining of MiniMax-Text-01 raised STEM, code, book and reasoning data to 70% (not drawn in the figure).

Read the top two rows together. Taking the categories as roughly comparable, MAI-Thinking-1's first phase raises STEM and math from 21.2% of MAI-Base-1's pretraining mix to 35% and holds code near 55%, so the extra share comes out of the rest, which falls from 24.3% to 10%. That is the forgetting side of the trade-off, and the replay section below asks how much general data is enough.

Synthetic reasoning traces

A reasoning trace is the step-by-step text a model writes before its answer (a chain of thought). A synthetic trace is written by another model, usually a strong reasoning model, for a question the lab collected. GLM-4.5, Z.ai's 355B-total, 32B-active MoE, gathered math, science and coding-competition questions from web pages and books and had a reasoning model write the reasoning, in a 500B-token stage. MiMo-VL-7B, Xiaomi's vision-language model, regenerated answers with long chains of thought and kept them only after rejection sampling (drawing several answers and keeping those that pass a quality check).

The reports give reasons, measured effects and some abstentions:

• Reasons given. Xiaomi's stated reason is that more data "with high-quality reasoning patterns" enhances a model's reasoning potential; chapter 03 gives the papers' argument that reinforcement learning (RL) mostly strengthens behavior the base model already produces. Ant gives a reason for the chain-of-thought corpora it adds in the last 600B tokens of Ling-1T's mid-training: they "pre-activate" reasoning, which Ant says raises the ceiling on reasoning and gives fine-tuning and RL a more stable foundation.
• Measured effects. During MiMo-VL's 550B-token stage 4, rich in long synthetic reasoning, scores rose by 9 points on MMMU, 14 on OSWorld-G and 16 on OlympiadBench, and the average MMMU answer grew from 680 to 2.5K tokens. OctoThinker, which midtrained Llama models before RL, found that long chain-of-thought data improved RL results but can make answers verbose and RL unstable.
• Labs that leave them out. MAI-Thinking-1 uses no synthetic data in midtraining and does not say why; MiniMax's M1 continued pretraining avoids it "strictly", also without a reason.

Replay: keeping general data in the mix

A network trained on a new distribution tends to lose what it learned on the old one: the same weights serve both, and nothing in the new batches pulls them back. This is catastrophic forgetting. In Ibrahim et al.'s continued pretraining of a 405M model, validation loss on the original Pile data rose from 2.17 to 3.56 after 200B tokens of German. The standard fix is replay, keeping a share of the old distribution in the new phase. With 25% replay the Pile loss ended at 2.33, while German loss moved only from 1.11 to 1.16. The authors find that even the lowest replay they tested, 1%, significantly reduces forgetting, while 50% slows learning of the new data.

Labs' replay shares vary widely with the stage:

• DeepSeek-Coder-V2, DeepSeek's 2024 code model, comes from what DeepSeek calls continued pre-training of an intermediate DeepSeek-V2 checkpoint, a phase name the boundary rule counts as midtraining. That phase ran 6T tokens, with 30% natural language sampled from DeepSeek-V2's corpus, and MiMo-7B cites it for its 70% math-and-code stage.
• MAI-Thinking-1 keeps 10% background data.
• Kimi-VL, Moonshot's vision-language model, replays 75% shorter data in its long-context stages (chapter 05).
• Step 3.5 Flash. StepFun gives replay in tokens: 81B of the 386B in its first mid-training stage (21%) and 10.5B of the 364B in its second (2.9%).
• Nemotron 3.5 Super. NVIDIA keeps 20% pretraining replay in its agentic stage and 90% in its Indic stage.

Muse Glimmer works through the learning rate instead: Phase 1 runs below the pretraining peak to balance synthetic-data gains "against acceptable regression on organic perplexity benchmarks".

Selecting documents and setting weights

A quality classifier is a small model that scores each document, for example for educational value or reasoning depth, so a lab can keep, drop or upweight it. Most labs describe such classifiers only for pretraining data. For midtraining, MAI is the most explicit: it keeps documents, mainly from its PDF corpus, that the Essential-Web taxonomy labels technically correct, at least intermediate in reasoning depth and at Bloom level "Analyze" or above (the fourth of six levels from Remember to Create). A fine-tuned 0.5B-parameter model produces those labels.

Weights come from small experiments. MAI tunes weights within each category with single-source microanneals, the short test anneals described in chapter 01. One OLMo 2 microanneal result bears on "how much": a 10/90 math-to-web mix lifted its GSM8K-subset score from 28.5 to 61, near the 63.5 of a 35/65 mix, so presence mattered more than share. MAI also caps each source's epochs in midtraining, more strictly for sources the model already seems to have memorized (chapter 09).

Put together, planning a phase is a loop over sources: turn each weight into tokens and epochs, and check the epochs against the source's cap.

# Plan one midtraining phase: tokens and epochs per source, with epoch caps.
def plan_phase(D_phase, w, U, seen, cap):
    # w[k]: mixture weight (sums to 1); U[k]: unique tokens in source k
    # seen[k]: epochs already used in pretraining; cap[k]: max total epochs
    assert abs(sum(w.values()) - 1.0) < 1e-6
    plan = {}
    for k in w:
        D_k = w[k] * D_phase              # tokens drawn from source k
        e_k = D_k / U[k]                  # passes over source k in this phase
        if seen[k] + e_k > cap[k]:
            raise ValueError(f"{k}: {seen[k] + e_k:.2f} epochs exceeds cap {cap[k]}")
        plan[k] = (D_k, e_k)
    return plan

def next_document(sources, w, rng):
    # each document is drawn from source k with probability w[k], then packed
    names = list(w)
    k = rng.choice(names, p=[w[n] for n in names])
    return sources[k].next_document()

"Overall, we further bias the mixture towards STEM, math, and code to build a strong foundation for reasoning RL climbs"

"This approach is expected to enhance specialized skills without compromising general language abilities"

1. Xiaomi, MiMo: Unlocking the Reasoning Potential of Language Model (MiMo-7B), 2025
2. Xiaomi, MiMo-V2-Flash technical report
3. Xiaomi, MiMo-VL technical report, 2025
4. Microsoft AI, MAI-Thinking-1 technical report, 2026
5. Moonshot AI, Muon is Scalable for LLM Training (Moonlight), 2025
6. Moonshot AI, Kimi-VL technical report, 2025
7. Z.ai, GLM-4.5 technical report, 2025
8. Meta, Muse Glimmer 30B: An On-Device Agentic Model, 2026
9. DeepSeek-AI, DeepSeek-Coder-V2, 2024
10. Muennighoff et al., Scaling Data-Constrained Language Models, 2023
11. Ibrahim et al., Simple and Scalable Strategies to Continually Pre-train Large Language Models, 2024
12. Team OLMo, 2 OLMo 2 Furious (microanneals), 2025
13. Essential AI, Essential-Web v1.0: 24T tokens of organized web data, 2025
14. Wang et al., OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling, 2025

03 Code and agent data

Why did 2025-26 midtraining start feeding models whole repositories, GitHub issue-and-fix histories and agent trajectories, and what does one such training sample look like?

This chapter asks two questions: what the data looks like, and why labs add it in midtraining rather than in pretraining or post-training. The data comes in three sample formats, each closer to the work a coding agent does:

• Repository-level code: a repository's related files in one sample, each definition before its use.
• Issue-PR bundles: a bug report, the relevant files and the fix, in one context.
• Agent trajectories: a model's reasoning, tool calls and tool outputs while it solves a task.

From files to repositories

Code corpora used to treat each source file as its own document, yet a function is often called in one file and defined in another, so the model rarely saw a definition next to its use. DeepSeek-Coder (2024) introduced repository-level samples, built in four steps:

1. parse which files import which, using regular expressions for import in Python, using in C# and include in C;
2. order the files so that each file's dependencies come before it;
3. put a comment with the file path at the top of each file;
4. concatenate each connected group of files into one training sample.

The format matters: training DeepSeek-Coder on file-level code only lowered cross-file completion in Java, TypeScript and C#.

import re
# Simplified from DeepSeek-Coder's Algorithm 1 (Python imports only; one sample
# per repository, where the original makes one per connected group of files).
IMPORT = re.compile(r"^\s*(?:from|import)\s+([\w.]+)", re.M)

def repo_sample(files):                       # files: {path: source text}
    module = {p[:-3].replace("/", "."): p for p in files if p.endswith(".py")}
    deps = {p: {module[m] for m in IMPORT.findall(src)
                if m in module and module[m] != p}
            for p, src in files.items()}      # files that p imports
    indeg = {p: len(deps[p]) for p in files}  # imported files not yet placed
    order, left = [], set(files)
    while left:
        p = min(left, key=lambda q: (indeg[q], q))  # minimal in-degree: cycles still finish
        order.append(p)
        left.remove(p)
        for q in left:
            if p in deps[q]:
                indeg[q] -= 1
    return "".join(f"# {p}\n{files[p]}\n" for p in order)  # path comment, then file

GLM-4.5, Z.ai's 355B-total, 32B-active mixture-of-experts (MoE) model, made this a named midtraining stage: 500B tokens of concatenated files from the same repository "to learn cross-file dependency". Kimi K2.5 (Moonshot's 1.04T-total, 32B-active MoE) has a joint vision-text stage, which Moonshot calls pre-training and these notes count as midtraining; it also upweighted repository-level code. MAI-Thinking-1 adds file-level documents alongside its repository format, treating the two as complementary tasks.

Issues, pull requests and commits

On GitHub an issue is a bug report or feature request, a pull request (PR) is the proposed change that resolves it, and a commit is one recorded change, shown as a diff of removed and added lines. SWE-bench (2023) turned that pairing into a benchmark: given a codebase and an issue, the model writes a patch that is evaluated against real tests. It has 2,294 problems from 12 Python repositories. An issue-PR pair is a worked instance of exactly that task.

GLM-4.5 put model-filtered issues, PRs and commits into its repository stage, with related ones concatenated into one context and commits in diff form. GLM-5 (744B total, 40B active) relaxed the repository filters, tightened filtering of individual issues and retrieved more relevant files per pair, reaching about 10 million issue-PR pairs and about 160B unique tokens. Kimi K2.5 upweighted issues, code reviews and commit histories to improve "agentic coding subtasks such as patch generation and unit test writing".

Figure 3.1 compares the three code formats on one made-up repository.

Agent trajectories

An agent trajectory is the record of a model working on a task with tools. GLM-5's report writes one as (q, r₁, a₁, o₁, …, r_n, a_n, o_n): the question, then for each round the model's reasoning r_i, its action a_i (a tool call such as a shell command or a file edit) and the observation o_i that the environment returns. ReAct (2022) introduced this interleaving of reasoning and actions. Written out with role markers, a trajectory is ordinary text and can be trained with next-token loss.

These labs describe such data in midtraining:

• Z.ai. GLM-4.5 added "large-scale synthetic agent trajectories" to its 100B-token 128K stage (filed under long context in chapter 00, since that is the stage's main purpose), and GLM-5 up-sampled them in its later, longer-context stages. GLM-5.2 "substantially expanded 1M-context training for coding-agent scenarios" but does not say whether that is mid- or post-training.
• DeepSeek. V4 (1.6T total, 49B active for Pro) grows its sequence length inside pretraining, but it also names an agentic mid-training phase of unstated size and says only that the agentic data strengthens coding (quoted below).
• Xiaomi. MiMo V2.6 (1.02T total, 42B active for Pro) builds its midtraining around realistic agent trajectories from coding, general, visual and research tasks, mixed with text, repository code, images, video and audio, first at 256K context (most of the compute) and then at 1M. It also adds self-correction data: Xiaomi took cases where MiMo had reward-hacked (earned reward by exploiting the environment or grader instead of doing the task) and had MiMo reflect on the faulty reasoning, rewrite the turn and continue properly. Xiaomi says these examples "improved the model's alignment".
• Meta. Muse Glimmer, a ~29.6B dense model for on-device agents, adds in its midtraining Phase 2 tool-use trajectories and demonstrations from a synthetic pipeline whose agent works in Docker containers that mimic email, calendar, messaging and code tools.
• StepFun and NVIDIA, among the labs added to the atlas later, describe agentic stages in detail. StepFun's first mid-training stage for Step 3.5 Flash (Feb 2026) emphasizes software-engineering and tool-use mixtures at 32K. NVIDIA's stage for Nemotron 3.5 Super (Oct 2026) runs 65.4B tokens at 256K and states its whole mix: 41% issue-PR pairs, 20% whole repositories, 20% pretraining replay, 10% agent traces, 5% long-context data, 3% commits and 1% build and CI logs.

Which tokens carry loss? A trajectory mixes text the model wrote with text the tools returned, so there are two conventions:

• All-token loss, plain next-token training, puts loss on every token, including tool output the model cannot predict.
• SFT-style loss trains only on what the model wrote. AgentTuning computes loss only on the model's output, and GLM-5's SFT keeps erroneous segments but masks them out of the loss.

Most labs here do not say which they use in midtraining. NVIDIA does: the agent traces are the only SFT-style data in its blend, and their tool responses are masked from the loss. Meta's synthetic pipeline, which feeds Muse Glimmer's midtraining Phase 2 and SFT alike, disables training on trajectory parts judged undesirable, such as bad tool calls. Figure 3.2 writes out one short coding trajectory and marks which rows each convention trains on.

Why midtraining: long samples need long sequences

The first reason is length. Repository and issue-PR samples are long, and GLM-4.5 and GLM-5 pretrain at 4K tokens. GLM-4.5 raised the sequence length from 4K to 32K to fit large repositories and used best-fit packing (placing whole documents into sequences instead of cutting them; chapter 05) to avoid truncating repo-level code. In GLM-5 an average issue-PR pair is about 16,000 tokens, four times the pretraining window, and every midtraining stage runs at 32K or longer (worked example below). In both releases midtraining is the first stage where such a sample fits whole.

Why midtraining: RL needs a model that sometimes succeeds

The second reason is the reinforcement learning (RL) that comes later. RL samples several attempts at a task and raises the probability of those that did better than the rest. If the model never solves a task, every attempt gets the same zero reward and nothing stands out.

GRPO, the group method GLM-5 says its RL builds on, makes this exact. It scores attempt i within a group of attempts at the same task by the advantage (r_i − mean r) / std r, where r_i is that attempt's reward. In a group of four where one attempt succeeds (rewards 1, 0, 0, 0), the mean is 0.25 and the standard deviation over the four is 0.43, so the success gets advantage +1.73 and each failure −0.58, and the update moves toward the success. When all rewards are equal, the numerator is zero for every attempt and std r is zero as well; the DAPO authors treat such a group's advantage as zero, giving zero policy gradient, and their method filters out groups whose accuracy is 0 or 1. Either way, RL learns from a task only once the model sometimes solves it.

pass@k measures "sometimes": the probability that at least one of k samples is correct. Small rates matter. With per-attempt success p and a group of 16 independent attempts (illustrative numbers), the chance of at least one success is 1 − (1 − p)¹⁶: 1 − 0.999¹⁶ = 1.6% at p = 0.001 and 1 − 0.98¹⁶ = 27.6% at p = 0.02. Only groups with mixed outcomes give GRPO a gradient, so raising p in midtraining multiplies the tasks RL can learn from.

Three papers support the argument. Yue et al. compared RL-trained models with their bases on math, code and visual reasoning. RL won at k = 1, but the base models scored higher at large k, and the RL models' reasoning paths were already in the base models' sampling distribution. Gandhi et al. traced what the base needs to habits such as checking work and backtracking: under identical RL on the Countdown arithmetic game, Qwen-2.5-3B reached about 60% and Llama-3.2-3B about 30%, and continued pretraining on OpenWebMath filtered for those habits let Llama match Qwen's improvement. OctoThinker reached a similar result with midtraining on math data.

The labs give shorter reasons:

• Xiaomi states the argument most directly (quoted below): agent midtraining "expands the exploration space" for RL.
• DeepSeek gives a narrower aim, to "enhance the coding capabilities".
• NVIDIA argues from placement: pretraining runs at short context and sees almost no agent interactions, while very long SFT can reduce what the model later gains from RL, so Nemotron 3.5 Super's agentic mid-training goes between them.

Where midtraining meets SFT

Midtraining used to mean more documents, and supervised fine-tuning (SFT) meant prompt-response pairs. The line is fading. GLM-4.5 already describes its midtraining data as medium-size, domain-specific and partly instruction data (chapter 00). Muse Glimmer feeds demonstrations from the same synthetic pipeline into midtraining Phase 2 and into SFT. MiMo-VL-7B, Xiaomi's vision-language model, lists "long-context Supervised Fine-Tuning (SFT)" as the fourth stage of its pre-training phase. What mostly separates the two is scale, since midtraining stages run hundreds of billions of tokens (500B in each of GLM-4.5's first two stages), yet MiMo-VL's "SFT" stage is 550B tokens, so even that line is blurring.

"an agent-centric mid-training phase that expands the exploration space for agentic tasks, enabling the model to discover more effective task-solving trajectories during post-training."

"we further enhance the coding capabilities of DeepSeek-V4 series by incorporating agentic data during the mid-training phase."

1. Guo et al., DeepSeek-Coder: When the Large Language Model Meets Programming, 2024
2. Z.ai, GLM-4.5 technical report, 2025
3. Z.ai, GLM-5 technical report, 2026
4. Z.ai, GLM-5.2 blog post
5. Moonshot AI, Kimi K2.5 technical report, 2026
6. Microsoft AI, MAI-Thinking-1 technical report, 2026
7. DeepSeek-AI, DeepSeek-V4 technical report, 2026
8. Xiaomi, MiMo-V2.6 technical report, 2026
9. Meta, Muse Glimmer 30B: An On-Device Agentic Model, 2026
10. Xiaomi, MiMo-VL technical report, 2025
11. Jimenez et al., SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, 2023
12. Yao et al., ReAct: Synergizing Reasoning and Acting in Language Models, 2022
13. Zeng et al., AgentTuning: Enabling Generalized Agent Abilities for LLMs, 2023
14. Chen et al., Evaluating Large Language Models Trained on Code (pass@k), 2021
15. Yue et al., Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?, 2025
16. Yu et al., DAPO: An Open-Source LLM Reinforcement Learning System at Scale, 2025
17. Gandhi et al., Cognitive Behaviors that Enable Self-Improving Reasoners, 2025
18. Wang et al., OctoThinker: Mid-training Incentivizes Reinforcement Learning Scaling, 2025

04 Stretching position encodings

A RoPE model trained on 4K-token sequences fails on a 128K-token input: which rotary dimensions break, and how do interpolation, a larger base, YaRN and position-free global layers each repair them?

What RoPE does to a query and a key

Rotary position embedding (RoPE; pretraining chapter 04 defines it) splits each query and key into pairs of numbers, treats each pair as an arrow in a plane, and turns pair i of the token at position m by the angle m·θ_i. In the dot product of a query at m and a key at n, position enters only through the difference (m − n)·θ_i, so attention sees how far apart the tokens are.

Each pair turns at its own speed θ_i, in radians per token; its wavelength λ_i counts the tokens per full turn. Fast pairs tell neighbouring tokens apart; slow pairs barely move over thousands of tokens. The speeds fall geometrically from a base b, counted over the d_r rotated dimensions of a head's d_h (many models rotate only part of the head; d_r = d_h when the whole head rotates):

A toy head with d_r = 4 and b = 10,000 has two pairs: pair 0 turns 1 radian (57°) per token, a full turn every 6.3 tokens, and pair 1 turns 10,000^−1/2 = 0.01 radian, a full turn every 628 tokens. A model trained at length L must later work at L′ = s·L. This chapter covers the position side; chapter 05 covers the data, stages and systems.

Why a model fails past its trained length

Train the toy head at L = 500. Pair 0 makes about 80 turns, so training shows it every angle; pair 1 reaches at most 5 radians (286°) and never completes a turn. At 2,000 tokens pair 1 reaches 20 radians, passing through angles from 286° to 360° that training never produced.

A real head splits the same way: fast pairs (λ_i ≪ L) have seen every angle, while slow pairs (λ_i > L) never finish a turn, so the network can read absolute position from them (the YaRN authors' argument) and past L they produce angles it never saw. LLaMA 7B, trained at 2,048 tokens, reached perplexity above 10³ at 4,096 (Chen et al.), and fine-tuning at the new length alone grew its effective window only from 2,048 to 2,560 tokens in more than 10,000 batches.

So every fix must keep the slow pairs within the trained angles without blurring the fast ones. On the toy head stretched to 2,000 tokens (s = 4):

Position interpolation: squeeze the positions

Position interpolation (PI; Chen et al., 2023) divides every position by s:

Chen et al. extended the context of LLaMA models from 7B to 65B parameters within 1,000 steps, the 7B and 13B to 32,768 tokens. The cost falls on the fast pairs: neighbouring tokens are now only θ_i/s apart on every pair, which removes RoPE's high-frequency detail, and the YaRN authors report that PI fine-tunes degraded beyond roughly s = 8 (Chen et al. themselves report usable 7B and 13B models at s = 16). Gemma 3 still uses PI, from 32K to 128K with a factor of 8, twice the length ratio.

Raising the base: ABF and NTK-aware scaling

A larger base b′ leaves pair 0 alone (θ₀ = 1 for any base) and stretches each pair more the slower it is:

Meta's Llama 2 Long work (Xiong et al., 2023) called this adjusted base frequency (ABF). It raised b from 10,000 to 500,000 for 400B tokens of continued pretraining. Without it the model could not attend beyond 4,000 to 6,000 tokens; with it, ABF alone held up across a 32,768-token retrieval probe.

The "NTK-aware" rule, written up in the YaRN paper, picks b′ so that the slowest pair (i = d_r/2 − 1) stretches exactly s times, as under PI:

For the toy head that is 10,000 × 4² = 160,000; for d_r = 128 and s = 32, b′ ≈ 338,000. Pair i is then stretched by λ′_i/λ_i = s^2i/(d_r−2), less than s for every pair but the slowest. Middle pairs whose wavelength is a little above L can therefore still meet new angles: the rule leaves some dimensions "slightly extrapolated", so s must be set above the length ratio.

YaRN: treat each pair by its wavelength

YaRN (Peng et al., 2023) decides per pair, by the number of turns r_i = L/λ_i that pair i makes within L. Pairs that turn many times keep their frequency, so nearby tokens stay distinct; pairs that never finish a turn are interpolated as in PI; a ramp γ mixes the two in between:

For LLaMA, α = 1 and β = 32 worked well. YaRN also divides the logits by a temperature t, which the authors link to attention entropy rising with length. For one head's query q_m ∈ ℝ^d_h at position m and key k_n ∈ ℝ^d_h at n:

The rule was fitted for lowest perplexity on LLaMA 7B to 65B. It multiplies every logit by (0.1 ln s + 1)², 1.30 at s = 4. Scaling q and k by √(1/t) inside the rotary embedding applies it without touching attention code.

def yarn_frequencies(d_r, b, L, s, alpha=1.0, beta=32.0):
    i = arange(d_r // 2)              # pair index, fast first
    theta = b ** (-2 * i / d_r)       # [d_r/2] frequencies
    r = L * theta / (2 * pi)          # r_i = L / lambda_i
    gamma = clip((r - alpha) / (beta - alpha), 0, 1)   # paper's ramp
    theta_new = (1 - gamma) * theta / s + gamma * theta
    scale = 0.1 * log(s) + 1          # sqrt(1/t)
    return theta_new, scale

def rope_yarn(x, pos, theta_new, scale):
    # x: [n, d_r] rotated part of queries or keys; pos: [n] positions m
    ang = pos[:, None] * theta_new[None, :]   # [n, d_r/2]
    x1, x2 = x[:, 0::2], x[:, 1::2]
    out = empty_like(x)
    out[:, 0::2] = x1 * cos(ang) - x2 * sin(ang)
    out[:, 1::2] = x1 * sin(ang) + x2 * cos(ang)
    return scale * out                # q.k gains scale**2 = 1/t

DeepSeek-V3 applies YaRN only to its 64-dimension decoupled rotary key (MLA, pretraining chapter 03), as DeepSeek-V2 did, during two 1,000-step stages at 32K and 128K. Of its 32 pairs, 11 are unchanged, 12 sit on the ramp and 9 are stretched 40× (Figure 4.1). The released code multiplies every logit by (0.1 ln 40 + 1)² ≈ 1.874, YaRN's 1/t. s = 40 over 4,096 positions covers 163,840, a margin over 128K.

What the labs chose

Labs pick b′ by hand and none says how; the table's last column gives the NTK-aware rule for comparison. Their choices range from about the rule (Kimi-VL, MiMo-V2-Flash) to 16 times above it (MiMo-7B).

MiMo-7B and Kimi-VL call these stages the end of pretraining. These notes count them as midtraining (chapter 00): each is a distinct stage after the main decay that changes the context length (clause 2 of the rule), and Kimi-VL's also continues training an earlier base, an intermediate Moonlight checkpoint (clause 3). Gemma 3 also extends its context at the end of pre-training, but its report names no separate phase and gives no learning-rate schedule, so the rule counts the extension as pretraining, like DeepSeek V4's growth inside its run.

2025-26: global layers without RoPE

Each fix above retunes RoPE whenever the context grows. Several 2025-26 models avoid that: they drop position encoding (NoPE) from global layers and keep RoPE, if at all, where offsets stay short. A window layer never sees an offset beyond its window, and a NoPE layer has no angles at all. Two designs appear:

• NoPE global layers between RoPE window layers. Llama 4's iRoPE interleaves "attention layers without positional embeddings" (global layers in Meta's reference code). MAI-Thinking-1 keeps RoPE (base 10,000) only on 512-token window layers and describes no rescaling for its 16K → 64K → 256K stages. Muse Glimmer, Meta's on-device agentic model (quoted below), extends from 32K to 128K in midtraining.
• NoPE attention, with other layers carrying order. Kimi Linear and Kimi K3 use NoPE in every MLA layer and let Kimi Delta Attention (KDA) layers, linear-attention layers with a fixed-size state (pretraining chapter 03), carry order; K3 grows from 8K to 1M "without any positional-encoding modification". In NVIDIA's Nemotron hybrids the Mamba layers "provide implicit positional information", so the few attention layers use no RoPE and there is nothing to rescale (chapter 05 shows how they extend).

Llama 4 also adds temperature scaling at inference, its counterpart to YaRN's temperature. With its attn_temperature_tuning flag on (off by default), Meta's reference code scales a NoPE layer's query at position m:

The factor is 1 while m + 1 < 8,192 and 1 + 0.1·ln 129 ≈ 1.49 at m + 1 = 1,048,576. The constants are the code's defaults, and the code cites Scalable-Softmax, which argues that softmax attention flattens as the number of keys grows. Llama 4's RoPE layers are still rescaled in one model: the code notes "Only Llama-4-Scout uses scaled rope", which divides the frequency of every pair with a wavelength above 8,192 tokens by 16.

Who uses it and why

"When extending the sequence length to 32K, we also adjusted RoPE’s base frequency from 10,000 to 1,000,000 for better long-context modeling ability."

"Second, it simplifies long-context training, as it obviates the need for RoPE parameter adjustments, such as frequency base tuning or methods like YaRN"

"Rotary position embeddings (RoPE) with θ [the base b] = 500,000 are applied exclusively to the local attention layers; the global layers rely on the accumulated positional signal from surrounding local layers, making the model extendable."

1. Chen et al., Extending Context Window of Large Language Models via Positional Interpolation (2023)
2. Xiong et al., Effective Long-Context Scaling of Foundation Models (ABF, 2023)
3. Peng et al., YaRN: Efficient Context Window Extension of Large Language Models (2023; also covers NTK-aware scaling)
4. Nakanishi, Scalable-Softmax Is Superior for Attention (2025)
5. DeepSeek-AI, DeepSeek-V2 Technical Report (YaRN target length)
6. DeepSeek-AI, DeepSeek-V3 Technical Report, config.json and modeling_deepseek.py
7. Z.ai, GLM-4.5 Technical Report, GLM-4.5 config.json, GLM-5 config.json, GLM-5.2 config.json, GLM-5.2 blog
8. Xiaomi, MiMo-7B Technical Report, MiMo-7B-Base config.json and MiMo-V2-Flash Technical Report
9. Kimi Team, Kimi-VL Technical Report and config.json; Kimi K2.5 Technical Report and config.json
10. Kimi Team, Kimi Linear Technical Report and Kimi K3 Technical Report
11. Gemma Team, Gemma 3 Technical Report
12. OpenAI, gpt-oss Model Card, config.json and reference model.py
13. Meta, The Llama 4 herd (blog), reference model.py and args.py
14. Meta, Muse Glimmer 30B technical report
15. Microsoft AI, MAI-Thinking-1 Technical Report

Lab RoPE scaling playground

When a model trained at length L is stretched to L′, which rotary dimensions see angles they never saw in training, and what does each method do about it?

This demo goes with chapter 04. RoPE rotates each pair of query and key dimensions at its own frequency θ_i = b^−2i/d_r (d_r: rotated dimensions per head), so pair i makes one full turn every λ_i = 2π/θ_i tokens. Each dot is one pair, placed at its wavelength after the method is applied; the dashed curve is the original wavelength, and the two horizontal lines mark L and L′. A pair whose wavelength is longer than L never completed a full turn in training, so at longer contexts it can reach angles the model has not seen. Pick a real model's settings or change them yourself. Three things to try:

1. Choose the DeepSeek-V3 preset, set the method to "none" and count the orange pairs; then switch back to YaRN.
2. Choose position interpolation with s = 32 and see how far even the fastest pair (pair 0) moves.
3. Compare the NTK-aware change and YaRN at s = 8: find the middle pairs that still reach unseen angles under NTK.

05 Long-context data, stages and systems

Why do labs reach 128K to 1M tokens of context in a few stages instead of training long from the start, and what data and parallelism does each long stage need?

Chapter 04 fixed the positions. Here a stage is a stretch of training at one sequence length L. A long stage raises three separate questions, taken in turn below: when and in how many steps the length grows (staging), what fills a sequence of 128K tokens or more (data), and how to train a sequence that outgrows one GPU (systems).

Staging: why grow the length in steps

Three pressures push long sequences late, and a fourth argues for small steps:

• Cost. Attention compares every pair of tokens, so its FLOPs grow with L² per sequence and with L per step at a fixed token batch. From 4K to 8K, each sequence's attention costs 4× as much, but a batch of equal tokens holds half as many sequences, so the step's attention cost doubles. MAI-Thinking-1 also cites "low MFU" (model FLOPs utilization) at long lengths.
• Scarcity. "genuinely long and coherent documents and videos are scarce relative to short text" (Kimi K3).
• Short-context quality. Kimi-VL replays 75% shorter data in its long stages to keep short-context ability, and NVIDIA trains 8% of Nemotron 3 Ultra's long-context iterations at 4K for the same reason.
• Stability. MiniMax made M1's extension smoother, four stages from 32K to 1M, because "excessively aggressive extensions of the training length can lead to a sudden gradient explosion".

A late stage costs little quality. In Xiong et al.'s 7B runs, switching from 4K to 32K at 80% of training used 2.27 × 10²² FLOPs against 3.78 × 10²² for 32K throughout. The long-document QA scores were mixed: NarrativeQA unchanged at 18.5 and QuALITY 38.3 against 37.9, but Qasper 25.0 against 28.6 and QMSum 11.0 against 11.5; the authors summarize it as "almost no loss on performance". In MAI's ablations, 1T tokens at 32K then about 100B at 256K matched the code loss at 128K of a full 1T-token run at 128K, and most of the gain came within the first 1–10% of the extension.

Staging: how the schedules changed

The reports show three patterns (Figure 5.1 draws eight schedules to scale):

• A few steps after the main run (through 2025 and in several 2026 releases). DeepSeek-V3 went 4K → 32K → 128K at the final pretraining learning rate, 0.84% of its 14.93T tokens. GLM-4.5 trained 1T at 32K and 100B at 128K; GLM-5 went 32K (1T) → 128K (500B) → 200K (50B); Kimi K2.5 trained 500B at 32K and 200B at 262,144. MAI-Thinking-1 went from 16K (30T) to 64K (3.4T) and 256K (150B), re-packing one midtraining mix, biased toward STEM, math and code, at both lengths (its Appendix B.5 says 140B for the last phase). MiniMax-Text-01 had already reached 1M in Jan 2025, in three stages of 300B, 32B and 26B tokens at 128K, 512K and 1M.
• One jump to the longest length. NVIDIA skips the ramp: each of its hybrid Mamba models gets one long-context phase that goes straight to its longest length, 512K for Nemotron Nano 2 even though its target was 128K (in tests on an 8B model, 512K worked better than 256K or 128K), and 1M for Nemotron 3 Super and Ultra (34B and 33B tokens).
• Growth inside the main run (new in 2026). DeepSeek V4 grows 4K → 16K → 64K → 1M inside one 32T to 33T run without per-length token counts. Under the rule this growth is pretraining, as DeepSeek also calls it, but DeepSeek also names an agentic mid-training phase whose size and place in training are not given (chapter 03). Kimi K3 goes 8K → 64K in pretraining and 256K → 1M in its cooldown, "a small fraction of the overall training budget", which these notes count as midtraining. MiMo V2.6 extends 32K → 256K partway through pretraining, then midtrains at 256K with the majority of its midtraining compute before a final 1M stage. DeepSeek V4.1-Flash trains from scratch at 64K and switches to 1M at 34T of 45T: 11T tokens (24%) at 1M. Its report names a decay stage and a context extension inside that run, so these notes count 28T–45T as midtraining.

Data: what fills a long sequence

The reports give three answers to what a long sequence should hold:

• Natural long documents, upsampled. GLM-4.5 "up-sample[s] long documents from the pre-training corpus"; Kimi K3 cleans long documents and videos and upsamples them so they are not "overwhelmed by short sequences during cooldown"; Kimi-VL fills 25% of each long stage with long text, video and documents.
• The same mix, re-packed. Xiong et al. found that reshaping the length mix gave no major benefit, and MAI, after trying upweighted long documents, found that "none of these modifications yielded meaningful improvements".
• Synthetic long-range tasks. Kimi K3 permutes and concatenates multimodal documents and sub-tasks so that tasks "can be solved only by attending to information scattered across the full 1M-token context". GLM-5, inspired by two published recipes for synthetic long-range data (NextLong and EntropyLong), interleaves highly similar texts "to mitigate the lost-in-the-middle phenomenon" (models use information in the middle of a long input worse than at its start or end). It adds data like MRCR (multi-round coreference resolution: reproduce one of several similar earlier replies in a long conversation) at 200K, and says the 200K stage "further bolstered the model’s performance even within the 128K context window".

Both GLM releases also add agent trajectories at long lengths (chapter 03).

Data: packing documents into sequences

Concatenating documents and cutting every L tokens splits any document that straddles a cut. Best-fit packing (Ding et al., 2024) avoids most cuts with the Best-Fit-Decreasing heuristic:

1. cut documents longer than L into pieces of at most L;
2. sort the pieces, longest first;
3. put each piece into the sequence with the least room that still fits it, or into a new one;
4. pad what is left.

Take L = 10 and documents of 6, 5, 4 and 3 tokens. Concatenate-and-cut gives [6 + 4 tokens of the 5] and [its last token + 4 + 3], splitting one document. Best-fit puts 6 and 5 in two sequences, then 4 into the first (the tightest fit) and 3 into the second: [6 + 4] and [5 + 3 + 2 padding], nothing cut.

def best_fit_pack(doc_lengths, L):
    chunks = []                    # 1. cut only docs longer than L
    for n in doc_lengths:
        while n > L:
            chunks.append(L); n -= L
        if n > 0:
            chunks.append(n)
    chunks.sort(reverse=True)      # 2. longest chunk first
    space, seqs = [], []           # free tokens per sequence
    for c in chunks:
        fits = [j for j, r in enumerate(space) if r >= c]
        if fits:                   # tightest sequence that fits
            j = min(fits, key=lambda j: space[j])
        else:                      # open a new sequence
            space.append(L); seqs.append([])
            j = len(space) - 1
        space[j] -= c
        seqs[j].append(c)
    return seqs                    # leftover space is padding
# Ding et al. find the tightest fit with a segment tree

GLM-4.5 skipped it in pretraining, where "random truncation is a good data-augmentation strategy", and used it in midtraining; DeepSeek V4.1-Flash reports a padding rate of at most 10⁻⁴. Packed documents can attend to each other unless the mask blocks it; DeepSeek-V3 packs but does "not incorporate cross-sample attention masking", and the other sources here do not say.

Systems: one sequence across many GPUs

At 256K to 1M tokens one sequence's activations outgrow a GPU, so context parallelism (CP) splits the sequence itself over P GPUs. Most of a layer treats each token alone and runs on each GPU's own tokens; attention is the exception, since every query needs keys and values held on other GPUs. Two designs move that data (Figure 5.2):

• Ring attention (Liu et al., 2023) keeps the queries in place and sends key-value blocks around a ring of GPUs, overlapping transfer with compute, so sequences can be "up to device count times longer".
• DeepSpeed-Ulysses (Jacobs et al., 2023) re-shards from tokens to heads with an all-to-all, so each GPU attends over the whole sequence for a subset of heads, and then shards back.

MAI-Thinking-1 used Ulysses-style CP only for long-context midtraining, adding ZeRO-3 (FSDP: parameters sharded and gathered just in time) for its 256K phase on 4,096 GB200 GPUs. Kimi K3 notes that softmax attention "requires ranks to exchange key–value blocks whose size grows with the sequence length", while its KDA layers (Kimi Delta Attention, linear attention with a fixed-size state; chapter 04) pass that state with one all-gather.

How labs check long context

• Retrieval. Needle-in-a-haystack (NIAH) hides a fact in long filler and asks for it: Kimi-VL recalls 100.0% up to 64K and 87.0% from 64K to 128K.
• Harder synthetic tasks. RULER adds multi-needle, multi-hop tracing and aggregation tasks; of models claiming at least 32K, only half held up at 32K. MRCR, described above, tests recall among deliberately similar replies. Gemma 3 27B (pretrained) scores 85.9 → 72.9 on RULER and 63.2 → 60.0 on MRCR from 32K to 128K.
• Loss probes. MAI measures loss on long inputs instead: models midtrained without the long stage "see their NLL increase once the contexts reach > 2x of the training distribution" (chapter 09 covers evaluation).

Who uses it and why

"Rather than training at the full 256K context length throughout pre-training or mid-training—which would be impractical due to the low MFU at long sequence lengths—we adopt a staged approach"

"Length alone, however, does not confer long-range capability."

"For datasets in mid-training, we applied best-fit packing to avoid truncating the reasoning process or repo-level code."

1. Xiong et al., Effective Long-Context Scaling of Foundation Models (2023)
2. Ding et al., Fewer Truncations Improve Language Modeling (best-fit packing, 2024)
3. Liu et al., Ring Attention with Blockwise Transformers for Near-Infinite Context (2023)
4. Jacobs et al., DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models (2023)
5. Hsieh et al., RULER: What's the Real Context Size of Your Long-Context Language Models? (2024)
6. Vodrahalli et al., Michelangelo: Long Context Evaluations Beyond Haystacks via Latent Structure Queries (MRCR, 2024)
7. Kaplan et al., Scaling Laws for Neural Language Models (2020)
8. DeepSeek-AI, DeepSeek-V3 Technical Report and config.json; DeepSeek-V4 Technical Report; DeepSeek-V4.1-Flash Technical Report
9. Z.ai, GLM-4.5 Technical Report and GLM-5 Technical Report
10. Kimi Team, Kimi-VL Technical Report, Kimi K2.5 Technical Report and Kimi K3 Technical Report
11. Microsoft AI, MAI-Thinking-1 Technical Report
12. Xiaomi, MiMo-V2.6 Technical Report and MiMo-V2-Flash Technical Report
13. Gemma Team, Gemma 3 Technical Report

06 Retrofitting architecture by continued training

How did DeepSeek and Z.ai give a trained dense-attention model sparse attention with at most 6.4% of its original pretraining tokens, and why did both train the new part alone first?

Why retrofit instead of retraining

Architecture ideas often arrive after a base model is finished, and retraining to adopt one repeats the whole run: 14.8T tokens for DeepSeek-V3, DeepSeek's 671B-total mixture-of-experts (MoE) model. A retrofit loads the trained weights, adds or changes a component, and keeps training until the network adapts. The labs call this continued pre-training; these notes count continued training that adds a new attention to an earlier base as midtraining (Chapter 00).

The sparse-attention retrofits in this chapter follow one sequence:

1. Swap. Load the trained base and add the new part with fresh weights.
2. Train the new part alone. Keep the old computation path, freeze the rest, and teach the new part to imitate what the trained model already does. This is short: 1,000 steps and 2 to 3 billion tokens. (GLM-5 does not say what it froze.)
3. Train together. Switch the new path on and train all the weights so the network adapts to it. This is the long part: 20B to 944B tokens.
4. Compare with the parent. Score both models on the same benchmarks, as bases or after identical post-training.

The main case is dense to sparse attention. Dense attention compares each query with every earlier token, so a sequence's attention cost grows with the square of its length. DeepSeek Sparse Attention (DSA), introduced in pretraining notes Chapter 03, lets each query read only a few thousand earlier tokens. The next sections walk DSA through the four steps; later ones show how other labs varied them.

The swap: what DSA adds to a trained layer

DSA leaves the trained attention weights in place and adds two things to every attention layer:

• A lightning indexer, a small scorer with its own projections, gives every earlier token a score for the current query.
• A top-k selector keeps the k highest-scoring tokens, and the existing multi-head latent attention (MLA, which caches one compressed latent per token) runs only over those.

With k = 2,048, a query at position 131,072, the end of a 128K sequence, has main attention read 2,048 latents instead of 131,072, one in 64; the indexer still scores all 131,072, at a much lower cost per pair. Up to position 2,048, top-2048 keeps every token, so selection only matters deeper into a sequence.

The indexer's score for query position t and earlier key position j sums over H_I indexer heads g. Each head matches a query vector computed from the query's hidden state x_t against a key vector computed from x_j, and a weight, also computed from x_t, scales the match:

DeepSeek-V3.2-Exp uses H_I = 64, d_I = 128 and k = 2048; GLM-5 uses 32 heads of 128. Per query–key pair the V3.2-Exp indexer does 64 × 128 = 8,192 multiply-adds in FP8, a ninth of the 73,728 that MLA spends scoring the same pair.

One fact shapes the rest of the recipe: top-k is a discrete choice. The language-modeling (LM) loss flows through attention over the selected tokens, but no gradient flows back through which tokens were selected. The indexer needs a loss of its own. Figure 6.1 shows the layer before and after the swap, and what trains in each of the two stages that follow.

The indexer's target: dense attention

A trained dense model already knows which tokens matter: its attention weights. For query t, DeepSeek sums the main attention weights a_t,h,j over heads h and normalizes over j. Each head's weights sum to 1, so the sum is H, the number of main heads, and the target is the head average. A Kullback–Leibler (KL) divergence pulls the softmax of the index scores toward it:

The loss is zero when the indexer's distribution matches the dense attention. Since top-k keeps the highest scores, an indexer trained this way keeps the tokens dense attention weighted most.

Training alone: the dense warm-up (stage 1)

The first stage keeps dense attention and freezes every parameter except the indexers, so the model's outputs do not change while the indexers learn ℒ_I from attention the model already computes. DeepSeek started from the DeepSeek-V3.1-Terminus base, already extended to 128K, and trained the indexers at learning rate 10⁻³ for 1,000 steps of 16 sequences of 128K tokens, 2.1B tokens.

DeepSeek gives the stage's purpose only as "to initialize the lightning indexer". Arithmetic shows what skipping it would mean. A new indexer ranks tokens almost at random, and if k of t earlier tokens are picked at random, each is kept with probability k/t, so the expected attention mass kept is k/t whatever the attention pattern: 2,048 / 131,072 = 1.6% at the end of a 128K sequence. The LM loss would then have to adapt the whole network to near-random selections.

Z.ai measured how far a warm-up alone gets. GLM-4.7-Flash, a 30B-total MLA model, scores 79.21 on RULER at 128K with dense attention. Switching to sparse attention after only a 1,000-step indexer warm-up gives 71.35, with shorter contexts "virtually unaffected", and 150B tokens of joint training recover 78.86. That recovery is the next stage's job.

Training together: sparse training (stage 2)

Then selection turns on and every parameter trains, with two losses on disjoint parameters:

• The main model trains on the LM loss alone.
• The indexer keeps ℒ_I, now restricted to the selected set S_t, which is all the main attention still scores. Its input is detached from the graph, so ℒ_I never reaches the main model.

This stage takes nearly all the tokens. DeepSeek used learning rate 7.3 × 10⁻⁶ and 15,000 steps of 480 sequences of 128K tokens, 943.7B tokens, on the data distribution of V3.1-Terminus's 128K extension. That learning rate is the one DeepSeek-V3 ended pretraining with (Chapter 08).

Compared with its parent after identical post-training, V3.2-Exp matched V3.1-Terminus on MMLU-Pro (85.0), scored 67.8 vs 68.4 on SWE-bench Verified, and scored four points higher on the independent AA-LCR long-context test in reasoning mode.

def indexer_kl(attn, scores):
    # attn:   [H, L, n] main attention weights over n candidates (n = L dense, k sparse)
    # scores: [L, n]    index scores I for the same candidates (masked entries = -inf)
    p = attn.mean(0)                          # [L, n] head average = normalized head sum
    q = softmax(scores, dim=-1)               # [L, n] indexer distribution
    return (xlogy(p, p) - xlogy(p, q)).sum()  # sum over t of KL(p_t || q_t)

# Stage 1, dense warm-up (V3.2-Exp: 1,000 steps x 16 sequences x 128K, LR 1e-3)
freeze(all_params_except_indexers)
for batch in data_128k:
    out = model(batch, sparse=False)          # dense attention: outputs unchanged
    loss = sum(indexer_kl(l.attn.detach(), l.scores) for l in out.layers)
    loss.backward(); opt.step()

# Stage 2, sparse training (15,000 steps x 480 sequences x 128K, LR 7.3e-6)
unfreeze(all_params)
for batch in data_128k:
    out = model(batch, sparse=True, k=2048)   # per layer: idx = topk(scores, k)
    loss_lm = cross_entropy(out.logits, batch.targets)   # reaches main model only
    loss_i = sum(indexer_kl(l.attn.detach(), l.scores.gather(-1, l.idx))
                 for l in out.layers)          # reaches indexers only (input detached)
    (loss_lm + loss_i).backward(); opt.step()

GLM-5 and Qwen: the same steps, other lengths

GLM-5, Z.ai's 744B-total MoE, pretrained with MLA on 27T tokens and mid-trained up to 200K context. Its DSA training "begins from the base model at the end of mid-training":

• Training alone. The warm-up ran 1,000 steps of 14 sequences of 202,752 tokens, the learning rate falling from 5 × 10⁻³ to 2 × 10⁻⁴.
• Training together. Sparse adaptation reused mid-training's data and hyperparameters for 20B tokens at a constant 1 × 10⁻⁵, where mid-training's linear decay ended.
• Comparing with the parent. Z.ai found this "enough to adapt the DSA model to match the performance of the original MLA model": at 128K the DSA base scored 86.0 vs 79.7 on SQuAD but 63.0 vs 66.3 on HotpotQA, and after the same SFT the two models "tie in training loss and evaluation benchmarks".

GLM-5's sparse stage is 47 times smaller than DeepSeek's, and neither lab explains the difference. Qwen ran the same two stages for Qwen3.8-Flash-Next (Aug 2026), a Gated DeltaNet hybrid with one full-attention layer in every four, to replace that full attention with Qwen Sparse Attention (QSA) during continued pretraining at 256K:

• Training alone. At first only the indexer trained, learning the dense attention by KL for 1,000 steps at η = 10⁻³ (about 2B tokens).
• Training together. Backbone and indexer then trained together for 8,000 steps at 2.5 × 10⁻⁵, about 200B tokens, between GLM-5's 20B and DeepSeek's 943.7B.

One indexer for four layers: GLM-5.2 IndexShare

The indexer still scores every earlier token, so its own cost grows with the context. For GLM-5.2's 1M context, Z.ai shares it: "every 4 transformer layers share a lightweight indexer", which sits in the first of the four, and the next three reuse its top-k indices. The released config gives 21 of 78 layers their own indexer, not 78 / 4 ≈ 20: the first two layers keep their own, and the other 76 form 19 groups of four, each led by one indexer, so 2 + 19 = 21. GLM-5.2 "is trained with IndexShare from mid-training with 128K sequence length"; the blog gives no token count, does not name the pretrained base, and does not say how the shared indexers were initialized.

The model card links Z.ai's IndexCache paper for IndexShare. Its training-aware variant trains each shared indexer to pick tokens that suit every layer reusing its choice: it distills the indexer of layer ℓ against itself and the m layers ℓ+1 … ℓ+m that reuse its indices, ℒ = (1/(m+1)) ∑_{ℓ′=ℓ…ℓ+m} ∑_t D_KL(p^(ℓ′)_t ‖ q^(ℓ)_t), which has the same gradient as one KL toward the average of their attention distributions. The blog does not say whether GLM-5.2 used this loss.

Other retrofits

The same steps recur when labs swap other parts:

• Cheaper attention, tested small. On GLM-9B, Z.ai tried cheaper swaps (sliding-window layers, Gated DeltaNet linear attention, and SimpleGDN, which reuses the pretrained query, key and value weights), each continued for 190B tokens with half the layers kept at full attention. All lost between 5.69 and 30.35 points on RULER at 128K.
• Cheaper attention at full scale. Ant's Ring-linear-2.0 (Sep 2025) converted Ling 2.0 bases to hybrid linear attention and continued them for 600B and 1T tokens. Ling 2.5 and 2.6 (Feb and Apr 2026) then turned the 20T-token Ling-2.0-1T into a 7:1 mix of Lightning linear-attention and MLA layers rather than retrain it (Ant's reason is quoted below). Ling 2.6's migration ran swap and warm-up twice in about 400B tokens (convert to Lightning, warm up the linear layers, convert the remaining layers to MLA, warm up MLA), followed by 8T tokens of continued pretraining, where switching straight to new high-quality data beat first replaying 2T tokens of the original corpus.
• A new objective. Muse Glimmer, Meta's ~29.6B dense on-device model, was pretrained on Muse Spark's top-256 logits with a KL-only loss for 95% of pretraining and cross-entropy for the rest. Mid-training starts from an averaged ("souped") checkpoint, the mean of the last four (Chapter 01), and Phase 2 uses cross-entropy for the reason quoted below. The report gives no token counts and does not state Phase 1's loss.
• A larger model's outputs. NVIDIA built both Nemotron 3.5 bases by continuing an existing Nemotron 3 base: each first trains a new MTP head for 100B tokens with the backbone frozen, much as the warm-ups above train the new indexer alone, and then trains the whole model for 1T tokens on the logits of the next larger Nemotron 3 base (Ultra for Super, Super for Lightning). Mistral's Ministral 3 models come from pruning Mistral Small 3.1 step by step and distilling each smaller model from it for 1 to 3T tokens, which Mistral calls pretraining and clause (3) counts as midtraining.

Later models made the switch earlier. DeepSeek moved it into pretraining: DeepSeek-V4-Flash used dense attention for its first 1T tokens, then warmed up the indexer and went sparse, and DeepSeek-V4.1-Flash trained sparse attention from scratch. Figure 6.2 puts the three DSA retrofits on one token axis, to scale.

Who uses it and why

"What makes DSA particularly interesting from a researcher’s perspective is how it was introduced via Continued Pre-Training from a dense base model. This avoided the “astronomical” cost of training from scratch."

"For the consideration of continued training from DeepSeek-V3.1-Terminus, we instantiate DSA based on MLA (DeepSeek-AI, 2024) for DeepSeek-V3.2-Exp."

"For this phase we have used a cross-entropy loss, as the distributional shift introduced by long-context and agentic data makes teacher logits less reliable at this stage."

"training a trillion-parameter replacement from scratch discards the 20T-token investment in Ling-2.0-1T."

1. DeepSeek-AI. DeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse Attention (2025), §1-2.
2. DeepSeek-AI. DeepSeek-V3.2 technical report (2025), §2.1-2.2.
3. DeepSeek-AI. DeepSeek-V3 Technical Report (2024), §4.
4. DeepSeek-AI. DeepSeek-V3.1 model card; DeepSeek-V3.2-Exp config.json.
5. DeepSeek-AI. DeepSeek-V4 technical report (2026), §4.2.2; DeepSeek-V4.1-Flash technical report (2026), §4.2.2.
6. Z.ai. GLM-5 technical report (2026), §2.1.1-2.1.2, §3.2, App. A; GLM-4.7-Flash model card.
7. Z.ai. GLM-5.2 blog, model card and config.json (2026).
8. Bai et al. IndexCache: Accelerating Sparse Attention via Cross-Layer Index Reuse (2026), §3.2.
9. Meta. Muse Glimmer: a 30B on-device agentic model (2026), §4.2.

07 Adding vision and audio to a language model

When a lab adds images and audio to a text model it has already trained, which parts does it train in which order, how many tokens does each stage take, and how does it check that the text skills survive?

Three parts and one risk

A vision-language model joins three parts:

• Encoder. A vision encoder, usually a ViT (a transformer over image patches), turns an image into feature vectors.
• Projector. Usually a two-layer MLP, it maps each feature vector to the language model's width d.
• Language model. The LM reads the projected vectors as soft tokens beside text.

Patches and token counts are in pretraining notes, ch 07; this chapter covers the stages that join the parts to an existing LM. Those stages all train on next-token loss over the text, so what tells them apart is which parts receive gradients and which stay frozen.

The risk sits in the projector. A new projector emits vectors the LM has never seen, and if the LM trains then, the updates land on weights that took trillions of tokens to learn. LLaVA (2023) set the order that avoids this: stage 1 trains only the projection on 595K image-caption pairs with encoder and LM frozen, "a compatible visual tokenizer for the frozen LLM"; stage 2 also updates the LM. Later labs grew this into a procedure of up to five stages:

0. Pretrain the encoder beside a small stand-in LM; the production LM is not attached.
1. Warm up the projector with encoder and LM frozen.
2. Align encoder and projector with the LM still frozen.
3. Train everything jointly, with text-only data mixed in.
4. Train everything at long context.

Labs skip or merge stages. Figure 7.1 shows which part trains in each stage and which releases report it.

Stage 0: train the encoder against a small LM

Kimi, DeepSeek and Xiaomi first train the encoder beside a small stand-in LM. Kimi-VL turned SigLIP-SO-400M (a contrastively trained image encoder) into its 400M MoonViT over 2T tokens; K2.5 trained MoonViT-3D for about 1T tokens against Moonlight-16B-A3B "with very few training FLOPs". DeepSeek's ViT saw about 47B image-text pairs, then 236B tokens through a 4B mixture-of-experts (MoE) LM that is then discarded. MiMo-ViT trained on more than 4T image tokens beside a small LM. These tokens never reach the production LM, so midtraining token counts leave them out.

Stages 1-2: align the projector, then the encoder, while the LM is frozen

MiMo-VL-7B (Xiaomi, on MiMo-7B-Base) runs both steps in full:

• Stage 1 freezes the Qwen2.5-ViT and the LM and trains only the randomly initialized projector on captions: 300B tokens at learning rate η = 1e-3.
• Stage 2 unfreezes the ViT and trains it with the projector, adding interleaved image-text pages to the captions, for 167B tokens, LM still frozen.

Other labs shorten or drop these steps:

• Qwen3-Omni (Sep 2025) dropped stage 2: it no longer trains encoder and adapter jointly on a frozen LLM, because the encoder may then compensate for the frozen LLM's limits and perceive worse, and trains each adapter before its encoder instead.
• Kimi-VL has no projector-only stage and uses one 0.1T stage that updates only MoonViT and the projector.
• K2.5 uses a "very short" projector-only stage to bridge to its 1T-parameter LM.
• MiMo-V2.5 (Xiaomi's April 2026 310B omni model, built on the text-only MiMo-V2-Flash backbone, with a 729M ViT and a 261M audio encoder) describes only a short-duration warm-up of its projectors.

Stage 3: joint training on mixed data

Then all parts train together, and text-only data stays in the mix. Three releases show the earlier size of this stage, 1.4T to 2T tokens:

• Kimi-VL loaded Moonlight (Moonshot's 16B-total MoE) after 5.2T text tokens, before its cooldown, and trained 1.4T more tokens, starting text-only and raising the multimodal share to at most 40%, then a 0.6T joint cooldown.
• MiMo-VL's stage 3 trains everything for 1.4T tokens at η = 1e-5, a hundredth of stage 1's rate, keeping "a limited amount of text-only data".
• GLM-V (GLM-4.1V-Thinking on GLM-4-9B-0414; GLM-4.5V and 4.6V on the 106B GLM-4.5-Air) runs 120,000 steps × 1,536 × 8,192, at most 1.51T tokens, without saying which parts train.

These stages have grown to pretraining size. K2.5 continues "from a near-end Kimi K2 checkpoint over additional 15T vision-text tokens at 4K sequence length". MiMo-V2.6 (September 2026, a 310B Flash and a 1.02T Pro; whether Flash reuses V2.5's base is not stated) trains its backbone on text, then attaches its ViT and audio encoder and trains "the full model on omni-modal data": 3T tokens for Pro, 22T for Flash. Both labs call this pre-training; the rule of chapter 00 counts continued training of an earlier base as midtraining. DeepSeek's V4-Flash-Vision-Exp card says only that visual modules were added to V4-Flash before continued training.

Stage 4: long multimodal context

The last stage extends the context with all parts training:

• Kimi-VL goes 8K → 32K → 128K over 0.3T tokens, raises the RoPE base b from 50,000 to 800,000, and mixes 25% long data with 75% replay of shorter data so the model can "learn long-context understanding while maintaining short-context ability".
• MiMo-VL's stage 4 runs 550B tokens at 32K and raises η to 2.5e-5 because "long-context packing significantly increases the effective batch size".
• K2.5 extends from 32K to 262,144 with YaRN, a RoPE rescaling (chapter 04, chapter 05).

STAGES = [  # MiMo-VL-7B, Table 1: (name, trainable parts, tokens, seq_len, lr)
    ("S1 projector warm-up", {"projector"},               300e9,  8192,  1e-3),
    ("S2 ViT + projector",   {"vit", "projector"},        167e9,  8192,  1e-4),  # table: "1e-4, 1e-5"
    ("S3 all parameters",    {"vit", "projector", "llm"}, 1.4e12, 8192,  1e-5),
    ("S4 long context",      {"vit", "projector", "llm"}, 550e9,  32768, 2.5e-5),
]

def run(model, data):
    for name, trainable, tokens, seq_len, lr in STAGES:
        for part in ("vit", "projector", "llm"):
            getattr(model, part).requires_grad_(part in trainable)  # frozen: no gradient
        opt = make_optimizer([p for p in model.parameters() if p.requires_grad], lr)
        seen = 0
        while seen < tokens:
            batch = data.sample(stage=name, seq_len=seq_len)      # mix changes per stage
            h_img = model.projector(model.vit(batch.images))      # (N, d) soft tokens
            loss = model.llm.next_token_loss(batch.text, h_img)   # loss on text positions
            loss.backward(); opt.step(); opt.zero_grad()
            seen += batch.num_tokens

Early or late, and how much vision

For K2.5, Kimi tested timing at a fixed vision-text budget: vision from 0% of training at 10:90 vision:text, from 50% at 20:80, or from 80% at 50:50. Early fusion scored best on text knowledge (45.5 against 43.9 and 43.1); with later injection, text scores dipped when vision arrived, then recovered (table in pretraining notes, ch 07). Starting from a near-end K2 checkpoint ruled out vision from 0%, so K2.5 took the other half of the lesson: a constant, moderate ratio (not given) from the start of its joint stage, rather than "aggressive vision-heavy training concentrated at the end". That stage begins after about 15T text-only tokens, so over K2 plus K2.5 vision arrives about halfway.

DeepSeek V4.1-Flash, a 552B-backbone MoE, has images in its data from the first of 45T tokens, at a 7:1 text-only to multimodal ratio. The encoder stays frozen (its final norm layer and the projector train) until the learning-rate decay stage at 28T, then trains with the LM "with a smaller learning rate". The stage rule counts that decay stage as midtraining. DeepSeek gives no reason for the timing.

Figure 7.2 puts these schedules on one axis: the tokens that pass through the LM before and during the multimodal stages.

Checking that text survives

Most labs report this in words. MiMo-VL says its models "maintain decent performance on text-only benchmarks" against MiMo-7B but prints no MiMo-7B column; GLM-V trains on pure text "to preserve its language capabilities" and shows no comparison. Two releases give numbers: K2.5's ablation, and the V4-Flash-Vision-Exp card, which beats V4-Flash-0731 on 6 of 7 text-agent benchmarks and trails on Cybergym, 75.3 to 76.7. Both models are post-trained, so this does not isolate the vision stage.

Who uses it and why

"This ensures the projector learns to map visual concepts to the language model's representation space effectively, providing informative gradient signals for subsequent training stages rather than noisy updates from a poorly aligned projector."

"To preserve text abilities, all stages that update the language model are joint training stages."

1. Liu et al., Visual Instruction Tuning (LLaVA), 2023
2. Kimi Team, Kimi-VL Technical Report
3. Kimi Team, Kimi K2.5 Technical Report
4. GLM-V Team, GLM-4.1V-Thinking and GLM-4.5V (also GLM-4.6V)
5. Xiaomi MiMo, MiMo-VL Technical Report
6. Xiaomi MiMo, MiMo-V2.5 model card
7. Xiaomi MiMo, MiMo-V2.6 Technical Report
8. DeepSeek-AI, DeepSeek-V4.1-Flash Technical Report
9. DeepSeek-AI, DeepSeek-V4-Flash-Vision-Exp model card
10. Pretraining notes, chapter 07: multimodal input

08 Changing optimizer, precision and hyperparameters mid-run

When a lab switches optimizer, turns on 4-bit training or changes a loss weight partway through a run, what state has to carry over, and what goes wrong if it restarts from zero?

What a checkpoint carries

A checkpoint holds more than the weights the model runs with. Labs change three kinds of state at stage boundaries, and each is risky to change mid-run for its own reason:

• Optimizer state. DeepSeek-V3 keeps FP32 master weights and gradients and stores AdamW's two moment estimates per parameter in BF16. These running averages took many steps to build; a new optimizer, or a restart, begins them at zero, and the first steps come out the wrong size or in a noisier direction.
• Number format. The precision the forward pass computes in. Rounding to fewer bits changes what the trained weights compute, and every later stage must keep the new format in its kernels.
• Schedule settings. The learning rate η, the batch size, loss weights, and the update rate of a mixture-of-experts (MoE) router's per-expert bias. A change takes effect at the next step for every weight it touches.

The sections below take the three in that order. Only the changes made for what follows midtraining (RL, serving in 4 bits, a hardware number format) come with stated reasons. Of the schedule changes, one keeps each step the same size when the context grows; the rest turn settings down as the run settles, at schedule boundaries. Some changes fall at token counts these notes count as main pretraining (in-schedule decay, Chapter 00); each is labeled. Figure 8.1 places several on a token axis.

Optimizer: why its state matters

AdamW, Adam with decoupled weight decay (pretraining notes Chapter 08), does not step by the raw gradient g. It keeps two running averages per weight, of g and of g², corrects them for starting at zero, and steps by their ratio:

The ratio is what makes warm Adam gentle: averaging noisy batches shrinks m̂ more than √v̂. For a weight whose gradient flips sign from batch to batch, m̂ nearly cancels while √v̂ does not, so the weight barely moves. Kimi's Moonlight report measured AdamW update RMS "usually around 0.2 to 0.4" (in units of η).

Restart m and v at zero and the averaging is gone. The first step has m̂₁ = g₁ and v̂₁ = g₁², so every weight moves by η · sign(g₁) and the update's root-mean-square (RMS) is η. A cold restart at an unchanged learning rate therefore takes first steps 2.5 to 5 times their usual size until the averages refill, over roughly 1/(1−β₂) = 20 steps for β₂ = 0.95.

Muon sets its step size by construction instead. It keeps one momentum matrix per weight matrix and replaces it with the nearest semi-orthogonal matrix, whose singular values are all 1 (computed by Newton–Schulz iteration, NS):

The step size comes from the orthogonalization, not from the state: a cold buffer (M₀ = 0) gives O₁ = NS(G₁), a normal-size step in a noisier single-batch direction, until momentum μ = 0.95 has averaged about 20 steps. Muon has no counterpart to Adam's v; its M plays the role of Adam's m.

Optimizer: switching AdamW to Muown before RL (MiMo-V2.6)

MiMo-V2.6-Pro, Xiaomi's 1.02T-total omni-modal MoE, was pretrained with AdamW on 27T text tokens and then 3T omni-modal tokens; the lab calls both pre-training, and these notes count the omni stage as midtraining. Its agent-centric mid-training, at 256K and then 1M context, switched the hidden weight matrices to Muown, a Muon variant, while embeddings, the LM head and the MoE router stayed on AdamW. The reason is the RL stage that follows: AdamW showed "diminishing optimization efficiency as the batch size increased in mixed-task RL", while Muon "retains stronger data efficiency beyond the critical-batch-size regime". The critical batch size is the point past which larger batches give diminishing returns (McCandlish et al.).

The switch is risky in two ways. The new optimizer's state starts cold, as above, and the new optimizer may not suit weights shaped by the old one: Qu et al. report that "naively switching to Muon for fine-tuning leads to degraded performance due to an optimizer mismatch", with damage that grows with update strength, and Moonlight found Muon SFT of an AdamW-pretrained Qwen2.5-7B only "on par" with AdamW. MiMo cites the risk and reports "no loss spike throughout the training process".

Muown targets a drift that Muon leaves unchecked. Its authors split a weight matrix's spectral norm (its largest singular value) into a row-magnitude factor and a row-coherence factor, and found that under Muon without weight decay the largest row norm grows and pulls the spectral norm up with it. Muown therefore gives the row magnitudes their own parameter, which MiMo describes as "explicit row-norm control, mitigating spectral-norm drift and reducing sensitivity to weight decay":

Here ρ holds the row magnitudes and ‖R‖_row the row lengths of the direction matrix R. What a switch carries over follows from the Muown paper's initialization; MiMo does not say how it initialized Muown or what learning rate mid-training used.

• The weights stay put. The paper starts from R = W and ρ = the row norms of W, so the weights do not move.
• The new state starts at zero. Muown keeps Muon momentum for R, ρ with a cached copy of ‖R‖_row, and Adam moments for ρ. R itself is rebuilt from W each step, so Muown adds no full-size state to Muon.
• AdamW's m and v for these matrices are dropped, since nothing in Muown uses them.

The paper scales the R step "to match the RMS norm of a typical Adam update", so one learning rate serves both parts. A switch can therefore start from AdamW-trained weights; the sketch below follows the paper's rule.

def switch_to_muown(ckpt, hidden):          # illustrative, per the Muown paper
    state = {}
    for name, W in ckpt.master_weights.items():  # FP32 weights W, unchanged
        if name in hidden:                      # attention, MLP and expert matrices
            rho = W.norm(dim=1)                 # ρ: [d_out] row magnitudes of W
            state[name] = dict(
                rho=rho,                        # ρ, updated by Adam
                R_rownorm=rho.clone(),          # cached ‖R‖_row (R = W here)
                M=zeros_like(W),                # Muon momentum for R, starts cold
                m_rho=zeros_like(rho),          # Adam first moment for ρ, starts cold
                v_rho=zeros_like(rho))          # Adam second moment for ρ, starts cold
            # each step rebuilds R = diag(R_rownorm / rho) W, so R = W at the switch;
            # AdamW's m and v for W are dropped: nothing in Muown uses them
        else:                                   # embeddings, LM head, MoE router
            state[name] = ckpt.adamw_state[name]  # stays on AdamW (carry-over assumed)
    return state

At the start of RL, MiMo carries over "the FP32 master weights and Muown's row state from the SFT checkpoint", to "stabilize MXFP4 training" (MXFP4 is the 4-bit format of the next section).

Precision: when 4-bit training starts

A model served in 4 bits computes with rounded weights, which is not quite the function it was trained as. Quantization-aware training (QAT) lets training see the rounding. It runs the forward pass on quantized weights while gradients update an FP32 master copy, passing through the rounding as if it were the identity (the straight-through estimator):

In MXFP4, blocks of 32 four-bit values share one power-of-two scale (pretraining notes Chapter 11). Each value is 0, ±0.5, ±1, ±1.5, ±2, ±3, ±4 or ±6 times the scale, so the grid is coarse: with δ = 0.25, a weight of 0.37 is 1.48δ and rounds to 1.5δ = 0.375, while 0.90 is 3.6δ and rounds to 4δ = 1.0, 11% too large. Turning QAT on changes every forward pass at once, and labs turn it on at different points:

• In mid-training. MiMo-V2.6 starts MXFP4 QAT in mid-training, "allowing the model to adapt to low-precision computation while preserving model quality", and its RL rollouts then run MXFP4 experts.
• From SFT through RL. Kimi K3 runs MXFP4-weight, MXFP8-activation QAT through all of post-training, SFT and RL alike.
• In post-training. DeepSeek-V4 adds MXFP4 QAT for expert weights and the indexer's query–key path in post-training, for "inference acceleration and reducing memory traffic".
• In SFT. GLM-5 applies INT4 QAT in SFT "to provide better accuracy at low-precision".

No lab compares start points. Starting earlier gives the model more tokens to adapt; every later stage must then keep the quantization in its training kernels.

Precision: hardware number formats

Low-precision formats store each block of numbers with a shared scale, so changing the scale format changes how every block is rounded, and the change has to reach every layer. DeepSeek-V3 trained in FP8 with one scale per 1×128 activation tile and per 128×128 weight block, and rounded only some scales to powers of 2. DeepSeek-V3.1 "is trained using the UE8M0 FP8 scale data format on both model weights and activations to ensure compatibility with microscaling data formats". UE8M0 is an unsigned, exponent-only 8-bit scale, the E8M0 scale format of the Microscaling (MX) standard, so every scale is a power of two. The card does not say which stage used it. V3.1's only training before SFT is its 839B-token context extension, so these notes count it as midtraining, an inference. One layer missed the change: the V3.1-Terminus card says the self_attn.o_proj parameters "do not conform to the UE8M0 FP8 scale data format".

NVIDIA tested a precision switch before the anneal, inside main pretraining under these notes' rule. Nemotron 3 Super trains most linear layers in 4-bit NVFP4 for all 25T pretraining tokens; promoting every tensor to MXFP8 1T tokens before the anneal improved the loss curve but not downstream accuracy, so NVIDIA kept NVFP4 to the end.

Hyperparameters: batch size when the context grows

A longer sequence holds more tokens, so a batch of a fixed number of sequences grows in tokens, and each step averages its gradient over more of them. DeepSeek-V3 and MiMo-V2-Flash both cut the batch, counted in sequences, by the factor the context grew, which keeps tokens per step fixed: 62.9M for DeepSeek-V3 at 4K, 32K and 128K (Chapter 05 works through these numbers and their GPU-hour cost) and 67.1M for MiMo-V2-Flash at 32K and 256K (worked example below). DeepSeek-V3 also kept its learning rate at the value pretraining ended on instead of restarting the schedule (quoted below), while MiMo-V2-Flash kept decaying it. Neither report gives the batch unit or a reason. MiMo-VL went the other way at its 32K stage and raised the learning rate because packing grew its effective batch (Chapter 07).

Hyperparameters: loss weights, balancing and frozen parts

Multi-token prediction (MTP) adds a head that predicts one further token, with its loss scaled by a weight (pretraining notes Chapter 06). Every report below lowers that weight from 0.3 to 0.1 late in the run:

• Inside main pretraining under these notes' rule: DeepSeek-V3 cut it from 0.3 to 0.1 after 10T of 14.8T tokens, GLM-4.5 after its first 15T, and DeepSeek-V4 at the start of LR decay.
• At a named late stage: MiMo-V2-Flash, Xiaomi's 309B-total MoE, cut it at its named mid-training, 0.3 in Stage 1, 0.1 in Stages 2 and 3, and MiniMax M2 annealed it from 0.3 to 0.1 during its named decay phase.

Auxiliary-loss-free balancing keeps experts evenly loaded with a per-expert routing bias c_e (for expert e) that is added to routing scores only when choosing experts. After each step, an overloaded expert's bias falls and an underloaded one's rises, by a fixed rate γ:

Setting γ = 0 freezes the biases at their learned values. The loss-free balancing paper found that a high fixed rate "deteriorates load balance in the later stage" of training, and the reports slow or stop balancing late:

• Inside pretraining here: DeepSeek-V3 set γ = 0 for its last 500B tokens and GLM-4.5 after 15T.
• In midtraining: Moonlight, Kimi's 16B-total MoE, set γ = 0 in its 500B-token cooldown, and MiMo-V2-Flash cut γ from 0.001 to 1 × 10⁻⁵ in Stage 3, its context extension.
• Router frozen. StepFun goes further in both of Step 3.5 Flash's mid-training stages: it freezes the router weights themselves, turns off its expert-parallel balance loss and fixes the MTP weight at 0.1.

A frozen part can join training mid-run. DeepSeek-V4.1-Flash kept its vision encoder frozen until its learning-rate decay stage (midtraining in these notes) began at 28T of 45T tokens, then trained it jointly "with a smaller learning rate" (Chapter 07).

No lab gives a reason for these changes. What they share is timing: each sits on a schedule boundary, either the start of the learning-rate decay (DeepSeek-V3's and DeepSeek-V4's MTP weight, V4.1-Flash's vision encoder), the final low-learning-rate tail (DeepSeek-V3's bias rate), or a named stage (GLM-4.5 at 15T, Moonlight's cooldown, MiMo-V2-Flash's Stages 2 and 3).

Who uses it and why

"We therefore switch to a variant of Muon optimizer, Muown, for the hidden weight matrices during mid-training to prepare the model for subsequent large-batch RL."

"We perform quantization-aware training (QAT) [50] throughout the entire post-training stage, covering both SFT and RL, so that the model adapts to quantization-induced precision loss."

"The learning rate for both phases is set to 7.3× 10−6, matching the final learning rate from the pre-training stage."

1. Kingma and Ba. Adam: A Method for Stochastic Optimization (2014), §2-3.
2. Lion et al. Muown: Row-Norm Control for Muon Optimization (2026), §1, §3, §5.
3. McCandlish et al. An Empirical Model of Large-Batch Training (2018), §5.
4. Qu, Huang and Horvath. Can Muon Fine-tune Adam-Pretrained Models? (2026).
5. Liu et al. (Moonshot AI). Muon is Scalable for LLM Training (Moonlight) (2025), §2.2, §3.3, §3.5.
6. Jacob et al. Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference (2017), §3; Bengio, Léonard and Courville. Estimating or Propagating Gradients Through Stochastic Neurons (2013).
7. Wang et al. Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts (2024).
8. Xiaomi. MiMo-V2.6 technical report (2026), §3, §5.1; MiMo-V2-Flash technical report (2025), §3.
9. DeepSeek-AI. DeepSeek-V3 Technical Report (2024), §3.3, §4.2-4.3; DeepSeek-V3.1 and V3.1-Terminus model cards.
10. DeepSeek-AI. DeepSeek-V4 technical report (2026), §4.2.2, §5.2.1; DeepSeek-V4.1-Flash technical report (2026), p.15, §4.2.2.
11. Kimi Team. Kimi K3 technical report (2026), §4.1.4.
12. Z.ai. GLM-4.5 technical report (2025), §2.4; GLM-5 technical report (2026), §2.4.3.

09 Measuring a midtrained base and deciding what to leave out

How do labs tell whether midtraining made the base model better, and which data do they deliberately keep out of it?

The chapter takes the two halves of that question in turn:

• Did the phase help? Labs score the base with few-shot prompts, with its loss on held-out text (per token or per byte), with pass@k, and before and after a single phase.
• What stays out? Benchmark text, sources the model has already memorized, and data a lab judges risky.

Few-shot scores on the base model

A base model, the checkpoint before supervised fine-tuning, does not follow instructions, so labs prompt it with a few solved examples and a new question to continue. DeepSeek compares V4.1-Flash-Base with its two V4 bases on 20 benchmarks in one setting (16 scored for all three; the 4 multimodal ones only for V4.1-Flash), each with a fixed number of examples (5 for MMLU-Pro, 0 for HumanEval). The answer is then scored in one of two ways:

• By likelihood. Multiple-choice tests are often scored by which option the model finds most likely, which MAI describes as "an NLL-based scoring scheme over logits for A, B, C, and D".
• By sampling. Other tests sample an answer and check it.

Small gaps are noise: DeepSeek treats scores within 0.3 "at the same level", and K2 averages GPQA-Diamond over eight runs "to mitigate the high variance".

Loss on held-out text

The MAI-Thinking-1 report (Microsoft AI; base model MAI-Base-1) argues against accuracy for pretraining work. It tracks nearly 40 benchmarks scored by negative log-likelihood (NLL): the model reads held-out text, such as vendor-written worked solutions or private code, and the score is its mean loss. NLL needs no sampling or judge, and teacher forcing (conditioning on the true prefix) keeps one formatting slip from compounding. Multiple-choice skill, MAI notes, "typically emerges only at surprisingly large scale". Five category scores are weighted into one target, with code at 0.5. NLL per token compares models only when they share a tokenizer, as MAI's ablations do.

Bits per byte

Tokenizers cut the same text into different numbers of tokens, so per-token losses measure differently sized pieces. Bits per byte (BPB) fixes the denominator. The Pile paper defines it from the mean loss ℓ in nats per token:

N_T is the number of tokens and N_B the number of UTF-8 bytes in the evaluation text (L_T, L_B in the paper); all are scalars. N_T·ℓ is the total loss in nats, dividing by ln 2 turns nats into bits, and dividing by N_B spreads the bits over bytes. Same text, same bytes: BPB puts models with different tokenizers on one scale.

DeepSeek-V3 reported BPB on the Pile test set. V4.1-Flash-Base "achieves lowest BPB on all tasks" against V4-Flash-Base and V4-Pro-Base on held-out internal documentation, code and academic material. MAI compares MAI-Base-1 with DeepSeek, Kimi and Gemma bases on four held-out tasks: lower BPB than models of similar active size, with DeepSeek V4 Pro best. Both note that APIs do not return log-likelihoods, so only open-weight models qualify.

Figure 9.1 shows why the denominator matters on one toy string: the tokenizer with the lower loss per token is not the better model of the text.

pass@k: what reinforcement learning can reach

Chapter 03 defines pass@k, the chance that at least one of k sampled answers is correct, and its unbiased estimator. Because reinforcement learning (RL) learns only from answers the model sometimes produces, labs use pass@k on the base as a ceiling for RL. Yue et al. (2025) found that RL-trained models beat their bases at small k while the bases win at large k, and they treat the base as an upper bound on what current RL elicits. MiMo-7B adopts the metric from Yue et al.; its base's lead over other bases, a 32B one included, widens as k grows. The GLM-V report plots pass@k on the non-multiple-choice part of MathVista and says base performance "arguably sets the upper bound for the final results after reinforcement learning".

Before and after one phase

To judge one phase, labs score checkpoints before and after it on the same held-out set, checking what the phase targets and what it should leave alone. MAI's small-scale ablations of context extension do this with code NLL: the last 16K tokens of a repository chunk are scored after 16K to 256K tokens of context. After extension the loss falls steadily as context grows, while models mid-trained at shorter lengths get worse beyond twice their training length. At 16K and 32K all five configurations reach about the same NLL: no short-context cost (chapter 05; OLMo 2's few-shot before/after is in chapter 01).

Keeping benchmarks out

Decontamination removes test items from training data. The labs here mostly match n-grams, runs of n consecutive words, between benchmarks and training text; they differ in n and in what they do with a hit:

• GPT-3 searched for 13-gram overlaps (13 consecutive lowercase words) between benchmarks and training data; a bug left some in, so OpenAI also scored clean subsets to measure the effect.
• DeepSeek dropped pretraining text containing any 10-gram from evaluation questions or solutions, about six million math texts alone, and warns that "the n-gram based decontamination method cannot prevent the paraphrase of testset".
• MAI removes huggingface.co and its mirrors, runs a 20-gram fuzzy deduplication at 80% similarity, calls its methods "imperfect" and builds private benchmarks.
• Xiaomi drops SFT queries with 16-gram overlap (MiMo-7B) and removes benchmark images by perceptual hash (MiMo-VL).

import re

def grams(text, g):                  # GPT-3: lowercase words, punctuation stripped
    words = re.sub(r"[^\w\s]", "", text.lower()).split()
    return {tuple(words[i:i + g]) for i in range(len(words) - g + 1)}

def build_index(benchmarks, g=10):   # DeepSeek: 10-grams of questions and solutions
    index = set()
    for item in benchmarks:
        index |= grams(item.question, g) | grams(item.solution, g)
    return index

def is_clean(doc, index, g=10):      # sketch: drop the whole document on one hit
    return not (grams(doc, g) & index)

index = build_index(benchmarks)
corpus = [d for d in corpus if is_clean(d, index)]

Capping what the model already memorized

MAI also limits how often midtraining repeats a source. Between two pretraining checkpoints, it measures what fraction of a source's validation-loss improvement comes from tokens predicted with near certainty (NLL below 0.01). The intuition: a token the model already predicts almost perfectly has little left to teach, so if most of a source's loss gain comes from such tokens, the model is learning to recite that source rather than to generalize. MAI reads a high fraction as "memorization or highly repeated structure" and gives such sources stricter epoch caps. In the notation here (the report gives it in words; I assume near-certainty at the later checkpoint):

Here V_j is the validation tokens of source j and ℓ_t⁽¹⁾, ℓ_t⁽²⁾ the losses of token t at the earlier and later checkpoint. For example (illustrative numbers), if a source's validation loss falls by 100 nats in total and 70 of them come from tokens now below 0.01, m_j = 0.7 and the source is capped harder than one at 0.1. The report gives no cap values.

Removing data on purpose

Kimi K3.1 Pro, Moonshot's 3T-class open-weight model, removed data for safety. As post-training scaled, its vulnerability-research scores rose; Kimi traced this to cyber content in the general midtraining corpus and cyber tasks in the RL mix. For the open-weight release it removed both and kept general coding tasks. On its xdev-bench-hard exploit tasks, a checkpoint without the removal solves 90.5% at pass@3 and the released model 81.0%. Kimi does not say how the data was found. The blog calls K3.1 Pro "a post-trained version of Kimi K3" yet edits its midtraining corpus, so these notes count it as having midtraining. MAI leaves out synthetic data: in its midtraining "no new or synthetic sources are introduced".

What the reports leave out

Chapter 00 counts which releases describe midtraining at all and finds 27 distinct recipes. Of those 27, 12 state their token count, 5 allow only an estimate and 10 give no total size. Four give a mix share as a number, and only MAI gives its whole mix by domain: 35% STEM and math, 55% code, 10% other. Outside the grid, NVIDIA gives the whole mix of Nemotron 3.5 Super's agentic stage (chapter 03), and Ant gives Ling-2.6-1T's final mid-training phase at 256K as 43% general, 42% reasoning and 15% agentic data.

Who uses it and why

"use Bits-Per-Byte (BPB) as the metric to guarantee fair comparison among models using different tokenizers."

"we cap data source exposure during mid-training to avoid overfitting sources that appear highly memorized by the end of pre-training."

"For the open-weight release of K3.1 Pro, we identified and removed cyber-related data from both training stages, while retaining the general-purpose coding tasks needed for model usefulness."

1. Gao et al., The Pile: An 800GB Dataset of Diverse Text for Language Modeling, 2020
2. Chen et al., Evaluating Large Language Models Trained on Code (Codex, pass@k), 2021
3. Yue et al., Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?, 2025
4. Brown et al., Language Models are Few-Shot Learners (GPT-3), 2020
5. DeepSeek-AI, DeepSeek-V3 Technical Report
6. DeepSeek-AI, DeepSeek-R1 (decontamination appendix)
7. DeepSeek-AI, DeepSeek-V4.1-Flash Technical Report
8. Microsoft AI, MAI-Thinking-1 Technical Report
9. Kimi Team, Kimi K2 Technical Report
10. Kimi Team, Kimi K2.5 Technical Report
11. Kimi, Introducing Kimi K3.1 Pro
12. Xiaomi MiMo, MiMo-7B Technical Report
13. Xiaomi MiMo, MiMo-VL Technical Report
14. GLM-V Team, GLM-4.1V-Thinking and GLM-4.5V
15. Kimi Team, Kimi-VL Technical Report

10 Putting it together: three midtraining recipes

When a lab rebuilds its midtraining from one release to the next, what changes, and how much of the run does it take?

Chapters 00 to 09 took one midtraining decision at a time. This chapter follows three labs across releases and reads each release the same way: which phases followed the main run, in what order, how many tokens they took, and what changed besides the data.

• DeepSeek kept midtraining small for most of 2025 and then folded it into one long run.
• Z.ai's GLM named a midtraining stage early and kept its share near 5%.
• Xiaomi's MiMo already spent a fifth to a quarter of its tokens late in training with MiMo-7B and V2-Flash.

Figure 10.1 draws each release's midtraining tokens on one scale, colored by what each phase was for.

DeepSeek: from a context patch to over a third of the run

1. V3: a context patch. Two short YaRN phases stretched the context from 4K to 128K (chapter 04) at the learning rate pretraining ended on (chapter 08): 126B tokens, 0.8% of the run.
2. V3.2-Exp: a retrofit. On the V3.1-Terminus base, DeepSeek Sparse Attention went in by the steps of chapter 06, the indexer trained alone and then everything together, in about 946B tokens.
3. V4: length inside the run. Length growth moved into the main run, so under the rule in these notes V4 has almost no separate midtraining, only a phase of agentic coding data whose size and place are not given (chapter 03 quotes DeepSeek on it).
4. V4.1-Flash: named late phases again. A learning-rate decay stage from 28T to 34T and a 1M-token context extension from 34T to 45T, during which the learning rate keeps falling until 40T and is then held; the vision encoder joins at the decay (chapter 07). Counted as midtraining, that is 17T of 45T.

That is the largest share of any DeepSeek release (Kimi K2.5's 51% in chapter 00 is larger). Part of the jump is labeling: DeepSeek-V3 also held its learning rate constant for most of its run and decayed it over the last 4.8T, but inside its main schedule, so the rule counts that as pretraining (chapter 00 puts V3 at 33% if it were counted). The long-context work did grow: 11T at 1M against V3's 126B.

GLM: a named stage that grew longer, then sparse

1. GLM-4.5: three named phases. Repo-level code, then synthetic reasoning, then long context and agent data, 1.1T tokens up to 128K. Z.ai named the stage mid-training because it trains on medium-size, domain-specific datasets, including instruction data, rather than large general corpora (chapter 00 quotes its definition).
2. GLM-5: the same shape, longer, then a retrofit. GLM-5 aimed the stage at reasoning, long context and agentic ability and stretched it to 200K. Its mix keeps repository-level code with issues and pull requests, and upsamples long documents and synthetic agent trajectories in the later stages. It then added DeepSeek Sparse Attention to the finished base by the steps of chapter 06, with far fewer tokens than DeepSeek had used.
3. GLM-5.2: sparse from the start. A shared sparse-attention indexer trains from the start of 128K midtraining.

On GLM-5's retrofit, Z.ai reported that this was enough:

"Although the training budget is much smaller than that of DeepSeek-V3.2 (943.7B tokens), we find that it is enough to adapt the DSA model to match the performance of the original MLA model."

MiMo: a late-heavy schedule from the start

1. MiMo-7B: math and code, then reasoning. Stage 2 raised math and code to about 70% (chapter 02); stage 3 added synthetic reasoning at 32K. The report's schedule sums to 23.8T and its text says "approximately 25 trillion tokens", so the 6T are about a quarter either way.
2. V2-Flash: code, then context. A code-centric "Mid-training" stage at 32K, then a "Context Extension" to 256K at whose boundary batch, RoPE base and balancing rate changed together (worked example in chapter 08).
3. V2.6: the setup changes too. An omni stage added vision and audio (chapter 07); agent-centric midtraining, at 256K and then 1M, switched the optimizer and began 4-bit training (chapter 08).

MiMo-7B's two late stages already followed the pattern GLM and DeepSeek reached later: shift the mix toward math and code, then add synthetic reasoning and longer context at a low learning rate. V2.6 widened the purpose of the stage from data to the whole training setup:

"Mid-training bridges this general foundation and subsequent large-scale RL by further developing the model's agentic abilities, extending long-context support, and adapting the optimization setup for large-batch training."

The optimizer switch is the clearest example. Xiaomi pretrained V2.6 with AdamW, found that its efficiency fell as RL batches grew, and switched to a Muon variant in midtraining so that the model would already be adapted when RL began (chapter 08).

What midtraining is for

Across the three labs, the choices of chapters 01 to 09 sort into five purposes (Figure 10.2).

1. DeepSeek-V3 report
2. DeepSeek-V3.2-Exp report
3. DeepSeek-V4 report
4. DeepSeek-V4.1 report
5. GLM-4.5 report
6. GLM-5 report
7. MiMo-7B report
8. MiMo-V2-Flash report
9. MiMo-V2.6 report

A Glossary

Short definitions of the terms used in these notes, with the chapter that explains each one.

ABF (adjusted base frequency)

Extending context by raising the RoPE base b, which stretches the wavelength of every rotary pair except the fastest, and then training at the longer length. GLM-4.5 raised its base from 10,000 to 1,000,000 at its 32K step. ch 04

Agent trajectory

The record of a model working on a task with tools: the task, the model's reasoning and tool calls, and the tool outputs it saw, written out as one training text. ch 03

Annealing

Training at a learning rate that decays toward zero, usually on upweighted high-quality data. Also used for short decay runs that test whether a dataset helps. ch 01

Best-fit packing

Placing whole documents into fixed-length training sequences so that as few as possible are cut, instead of concatenating everything and splitting at fixed points. ch 03, ch 05

Bits per byte

A model's loss converted from nats per token to bits per byte of raw text, so models with different tokenizers can be compared on the same text. ch 09

Catastrophic forgetting

A model losing skills it had when training moves to a narrower data distribution. Midtraining guards against it with replay. ch 02

Checkpoint averaging (souping)

Averaging the weights of several checkpoints from the end of a run to get a smoother starting point; Muse Glimmer starts its midtraining from such an average, and Ling uses one in place of a learning-rate decay (WSM). ch 01

Coded releases

The 68 releases, from the labs these notes started with, in the cross-lab table behind the atlas's midtraining section (the six labs the atlas added later are not coded), each coded for its phases, tokens, context and data under one boundary rule. Releases that reuse a parent's phases count once as a recipe, giving 27 distinct midtraining recipes. ch 00

Context parallelism

Splitting one long sequence across several GPUs so its activations fit; ring attention passes key-value blocks between the GPUs while they compute. ch 05

Cooldown

Another lab name for the final phase in which the learning rate decays (Moonlight, Kimi K3). ch 00

Decontamination

Removing training documents that overlap with benchmark questions, usually by matching long n-grams, so evaluation scores are not inflated by memorization. ch 09

Dense warm-up

The first stage of a sparse-attention retrofit: the rest of the model is frozen and only the new indexer trains, to match the dense attention pattern with a KL loss. ch 06

DSA (DeepSeek Sparse Attention)

Attention in which a small lightning indexer scores earlier tokens and each query attends only to the top-scoring ones (2,048 in DeepSeek V3.2). DeepSeek and GLM added it to trained models by continued training. ch 06

Epoch

One full pass over a data source. Upweighting a small source in a long phase means repeating it several times. ch 02

Few-shot evaluation

Scoring a base model on a benchmark by putting a few solved examples in the prompt; labs keep the setting fixed so base models can be compared. ch 09

GRPO

Group relative policy optimization: RL that scores each of several attempts at the same task against the group's mean reward, so a task gives no signal until the model sometimes solves it. ch 03, posttraining ch 03

Held-out loss

The model's next-token loss on text it was not trained on, by domain; MAI prefers it to benchmark accuracy for judging pretraining and midtraining changes. ch 09

IndexShare

GLM-5.2's change to DSA: one indexer is shared by every four attention layers, trained from the start of midtraining. ch 06

Issue–PR data

Training samples built from a GitHub issue and the pull request that fixed it, showing a problem description followed by a code change. GLM-5 used about 10 million such pairs. ch 03

KDA (Kimi Delta Attention)

The linear-attention layer in Kimi Linear and Kimi K3, which keeps a fixed-size state instead of a growing cache and carries word order in place of RoPE. ch 04, pretraining ch 03

Learning-rate area

The sum of the learning rate over a stretch of steps. In the one-parameter model of chapter 01, a batch's influence on the final weights shrinks with the learning-rate area still to come after it, so the last data count most. ch 01

Lightning indexer

The scorer DSA adds to each attention layer, with its own small projections and scoring heads (64 heads of 128 dimensions in DeepSeek V3.2-Exp, computed in FP8), that picks which earlier tokens each query attends to. ch 06

MFU (model FLOPs utilization)

The share of a chip's peak arithmetic a training run actually uses; it falls at long context, which MAI cites as one cost of long stages. ch 05

Microanneal

A short annealing run on a candidate data source mixed with general data, used to test the source cheaply (OLMo 2). ch 01

Midtraining

In these notes: (1) any phase a lab names mid-training, annealing, cooldown, learning-rate decay stage, context extension or continued pre-training; (2) a distinct stage after the main decay that changes the data mix or the context length; (3) continued training of an earlier base (a released model or an intermediate checkpoint). It ends where supervised fine-tuning begins. ch 00

Mixture weight (w_k)

The share of a phase's tokens drawn from data source k. ch 02

MLA (multi-head latent attention)

DeepSeek's attention, also used by Kimi and GLM-5, which caches one compressed latent vector per token plus a small rotary key instead of full keys and values. ch 06, pretraining ch 03

Moonlight

Moonshot's 16B-total mixture-of-experts model from February 2025, trained with Muon, whose final phase the lab calls a cooldown. ch 00

MRCR, RULER, needle in a haystack

Long-context tests: find a planted fact in a long input (needle), harder retrieval, tracing and aggregation tasks (RULER), and recall of one of several similar earlier requests (MRCR). ch 05

Multi-token prediction (MTP) weight

The factor that scales the loss of the extra heads predicting later tokens; several labs lower it late in training, for example from 0.3 to 0.1. ch 08

Muon, Muown

Muon is an optimizer that orthogonalizes each weight matrix's update (pretraining notes chapter 08). Muown is a Muon variant by Lion et al. (2026) with explicit control of row norms; MiMo V2.6 switched to it from AdamW in midtraining to prepare for large-batch RL. ch 08

MXFP4

A 4-bit floating-point format in which each block of 32 values shares one power-of-two scale. MiMo V2.6 began MXFP4 quantization-aware training in midtraining. ch 08, pretraining ch 11

NoPE

No position encoding: attention layers without RoPE. Using it in global layers (Llama 4, Kimi Linear, Kimi K3, MAI-Thinking-1) avoids rescaling RoPE when context grows. ch 04

NTK-aware scaling

Raising the RoPE base by s^{d_r/(d_r−2)} so the fastest pairs stay almost unchanged and the slowest is stretched by exactly s. ch 04

Omni stage

MiMo V2.6's stage that attaches vision and audio encoders to the text model and trains everything on mixed-modality data: 3T tokens for Pro, 22T for Flash. ch 07

Optimizer state

What the optimizer keeps between steps besides the weights: Adam's running averages of the gradient and its square, Muon's momentum. Switching optimizer mid-run means starting this state again. ch 08

pass@k

The probability that at least one of k samples solves a task. Used as a rough ceiling for what RL can reach from a base model. ch 03, ch 09

Position interpolation (PI)

Dividing every position by the scale factor s before applying RoPE, so a longer input uses only angles seen in training; nearby tokens become harder to tell apart. ch 04

Projector

The small network, usually an MLP, that maps a vision or audio encoder's outputs into the language model's embedding space. Labs often train it alone first. ch 07

QAT (quantization-aware training)

Training with the rounding of the low-precision serving format applied in the forward pass. MiMo V2.6 starts MXFP4 QAT in midtraining; other labs start it at SFT or in posttraining. ch 08

Replay

Keeping a share of the earlier, general data in a specialized phase so the model does not forget it. ch 02

Repository-level code

Training samples that concatenate files from one repository, so the model learns how code depends on other files. ch 03

Retrofit

Changing a trained model's architecture, such as replacing dense attention with sparse attention, by continued training instead of training a new model. ch 06

River valley

A picture of the loss surface with steep walls and a flat floor: at a high learning rate the weights bounce between the walls, and decaying it lets them settle onto the floor, which is why loss drops quickly during decay. ch 01

RoPE wavelength (λ_i)

The number of tokens over which rotary pair i makes one full turn, 2πb^2i/d_r, where d_r is the number of rotated dimensions per head. Pairs whose wavelength exceeds the trained length never completed a turn in training. ch 04

Synthetic reasoning traces

Step-by-step solutions generated by a model, filtered for correctness, and mixed into late training to prepare the base for reasoning RL. ch 02

UE8M0

An 8-bit scale format that stores only an exponent (a power of two) for each block of FP8 values; DeepSeek-V3.1 trained with it for compatibility with microscaling formats. ch 08

Ulysses (sequence parallelism)

A way to split one long sequence across GPUs: an all-to-all exchange moves each GPU from holding some tokens for all heads to all tokens for some heads, and back. MAI-Thinking-1 used it for long-context midtraining. ch 05

Upweighting factor (u_k)

How many times larger a source's share is in a midtraining phase than in pretraining: u_k = w_k^mid / w_k^pre. ch 02

Warmup-stable-decay (WSD)

A learning-rate schedule held constant for most of training and decayed only at the end, which makes a separate decay phase on chosen data possible. ch 01

YaRN

A RoPE extension that leaves fast pairs unchanged, interpolates slow pairs by s, blends the pairs in between, and slightly sharpens attention with a temperature. DeepSeek-V3 used it with s = 40. ch 04

B Citation

To give credit if you find this work useful, please cite:

@misc{ma2026openrecipesmidtrain,
  title={Reading Open-Weight Recipes, Part 2: What 2025--26 Model Reports Disclose About Midtraining},
  author={Ma, Ziqiao},
  howpublished={\url{https://mars-tin.github.io/blogs/posts/open_recipes_midtraining}},
  note={Blog post},
  year={2026}
}