00 What posttraining does and what it costs

What can a pretrained base model not do on its own, which three tools do labs use to fix that, and how did the order and cost of those tools change between DeepSeek-V3 in December 2024 and Inkling in July 2026?

What a base model can and cannot do

A base model is the network that comes out of pretraining (and often a mid-training phase that reweights the data, for example toward math and code). It has learned one task, predicting the next token of text like its training data, and it will continue any text it is given. Posttraining is everything a lab does afterwards to turn that predictor into an assistant or agent that answers in a chat format, follows instructions, reasons before answering, calls tools and declines harmful requests.

Continuing text and helping a user are different goals. The InstructGPT paper (OpenAI, 2022) notes that base models often make up facts, produce biased or toxic text, or ignore the instruction. The skill is often present but unused, and two results show the size of the gap. Human raters preferred outputs of a 1.3B-parameter posttrained model to those of the 175B GPT-3. And DeepSeek-R1-Zero, trained with RL directly from the DeepSeek-V3 base, raised its single-attempt accuracy (pass@1) on the AIME 2024 math competition from 15.6% to 77.9%.

The shape of a 2025-26 pipeline

Most pipelines in these reports are assembled from four stages. The list is a composite, not one lab's recipe, and some labs skip a stage; Figure 0.1 below draws four real ones.

1. A short SFT. Fine-tune the base model on example conversations so it learns the chat format, when to stop and a readable layout for its reasoning. This is the cheap stage: one forward and backward pass per token of a small dataset.
2. RL on checkable tasks. The model answers prompts; a rule, test suite or judge model scores each answer; training favors the high scorers. Each step must generate answers token by token before it can learn, so sampling dominates the cost, and rewards and environments must be built first.
3. Specialists. Labs often run stage 2 separately per domain (math, code, agents, helpfulness) and pay its cost once per specialist.
4. Consolidation. One model absorbs the specialists, by SFT on their responses or by on-policy distillation, where it writes and the specialists grade every token; the latter adds a teacher pass over every student token. Some pipelines end with more RL on the merged model.

Three tools

Each stage uses one of three tools. They differ in who writes the response y that the model learns from, and in what tells the model whether y is good:

Thinking Machines labels the three as off-policy and dense (SFT), on-policy and sparse (RL), and on-policy and dense (OPD): on-policy means the trained model wrote y, dense means a signal at every token. Each tool is a loss on that model, the policy π_θ, a probability distribution over responses y to a prompt x.

Supervised fine-tuning (SFT) trains on fixed prompt-response pairs (demonstrations) from a dataset D, with the pretraining loss on the response tokens (chapter 01):

SFT imitates: it puts probability on whatever the demonstrations contain, mistakes included. Formally, write p_D for the distribution of demonstrations. Since E_pD[−log π_θ] = KL(p_D ‖ π_θ) + H(p_D), and the entropy H(p_D) does not depend on θ, minimizing the SFT loss minimizes the Kullback-Leibler (KL) divergence from the data to the model.

Reinforcement learning (RL) lets the model write its own responses and scores each with a reward r(x, y) from a rule, a test suite or a learned judge. It maximizes expected reward minus a penalty, weighted by β, that keeps π_θ near a reference policy π_ref (chapters 03 to 06):

InstructGPT used this form with a per-token KL penalty toward the SFT model. The update behind it: sample a response and push its log-probability up or down by how much its reward beat the average (the baseline). If the model answers a prompt correctly 30% of the time, with reward 1 or 0, the baseline is 0.3, so a correct sample gets weight 1 − 0.3 = 0.7 and a wrong one 0 − 0.3 = −0.3. RL is on-policy (it trains on the model's own samples), so it can find answers no demonstrator wrote, but it gets one number per response.

Distillation trains a student π_S to match a teacher π_T, an idea Hinton et al. (2015) introduced as using a large model's probabilities as "soft targets". The student is the policy being trained, so π_S = π_θ. Off-policy distillation runs SFT on teacher samples. On-policy distillation (OPD) samples from the student and penalizes, at every token, the gap between the student's and teacher's next-token distributions. DeepSeek-V4 merges several teachers this way, with a teacher weight w_i for teacher i (written here in the full-vocabulary form DeepSeek-V4 uses; most labs use only the sampled token's log-ratio, chapter 07):

The order inside the KL flips. SFT minimizes the forward KL, KL(data ‖ model); OPD minimizes the reverse KL, KL(student ‖ teacher), averaged over the student's own text. Chapter 07 shows why the direction matters.

# One schematic update per tool. logp(m, x, y) = sum over t of log m(y_t | x, y_<t)
def sft_loss(policy, x, y_demo):              # y written by people or another model
    return -logp(policy, x, y_demo)           # imitate the demonstration

def rl_loss(policy, x, reward, baseline):
    y = sample(policy, x)                     # y written by the model itself
    r = reward(x, y)                          # one number for the whole response
    return -(r - baseline) * logp(policy, x, y)   # policy-gradient surrogate, r constant

def opd_loss(student, teacher, x):
    y = sample(student, x)                    # on-policy: the student writes y
    loss = 0.0
    for t in range(len(y)):                   # the teacher grades every token
        p_s = next_token_probs(student, x, y[:t])   # shape [|𝒱|]
        p_t = next_token_probs(teacher, x, y[:t])   # shape [|𝒱|], teacher frozen
        loss += (p_s * (log(p_s) - log(p_t))).sum() # reverse KL over the vocabulary 𝒱
    # full-vocabulary form (DeepSeek-V4); most labs use only the sampled token (ch. 07)
    return loss

How pipelines changed

2022: SFT, a reward model, then RL. InstructGPT set the baseline: SFT on demonstrations for about 13k prompts, a reward model trained on labelers' rankings, then RL against it with PPO (Proximal Policy Optimization, chapter 03).

January 2025: RL at the center. DeepSeek-R1's sibling R1-Zero ran RL on the V3 base with rule-based rewards and learned long chains of thought (CoT, written reasoning before the answer) that were hard to read and mixed languages. R1 alternates tools instead (Figure 0.1, top row):

1. SFT on thousands of curated long CoTs (a "cold start");
2. reasoning RL with GRPO (Group Relative Policy Optimization), which replaces the learned critic with group scores: a critic is a second network that predicts the expected reward, and GRPO uses the average reward of several samples for the same prompt instead (chapter 03);
3. rejection sampling (keep only sampled responses that pass a check) for reasoning data, which with non-reasoning data reused from DeepSeek-V3 makes about 800K examples for a second SFT, restarted from V3-Base;
4. 1,700 steps of RL on all scenarios.

2024-26: specialists merged into one model. DeepSeek-V3 (December 2024) already generated reasoning SFT data with per-domain expert models. V3.2 (December 2025) made this the pipeline: RL-trained specialists for six domains, SFT of one model on their outputs, then one mixed RL stage, chosen because it avoids "catastrophic forgetting issues commonly associated with multi-stage training paradigms" (later stages erasing earlier skills). V4 (April 2026) kept the specialists and replaced the mixed RL stage with on-policy distillation from more than ten teachers.

Multi-teacher OPD appears in 12 of the 47 pipelines in the coded grid for the first seven labs, first in Xiaomi's MiMo-V2-Flash (December 2025), and, among the six labs added later, in NVIDIA's Nemotron 3 Ultra and Nemotron 3.5 Super; the coded grid is the release-by-feature table, checked against the labs' sources, behind the posttraining section of the atlas.

2026: RL first, SFT to merge. Microsoft AI's MAI-Thinking-1 (June 2026) runs no SFT before RL. Three specialist RL "climbs" start from the mid-trained base. When a climb collapses (its scores drop suddenly, for example after small numerical mismatches between inference and training build up), a mid-trained checkpoint is fine-tuned on the climb's own successful traces and RL restarts from it (self-distillation). A consolidation SFT merges the three teachers, and a light final RL stage tunes safety and style. Thinking Machines' Inkling (July 2026) is shortest: a bootstrap SFT on data from other labs' open-weights models, including Kimi K2.5, then asynchronous RL (rollout workers keep generating while the trainer updates, chapter 06) over math, code, tools, audio, image, chat and safety, scaled to over 30M rollouts (sampled responses).

Across these pipelines SFT's role narrowed to a short start or a step that merges RL-trained skills into one model, while RL and teacher-matching do most of the training.

What it costs

Stated posttraining compute rose fast, though the reports measure it unevenly:

• DeepSeek-V3 (December 2024): 5K H800 GPU-hours (one GPU for one hour) of posttraining.
• DeepSeek-R1, posttrained from the same base a month later: 147K.
• DeepSeek-V3.2 (December 2025): a budget "exceeding 10% of the pre-training cost".
• Kimi K3.1 Pro (October 2026): "a post-trained version of Kimi K3, with substantially larger post-training compute".
• Inkling gives no GPU-hours but says most of its posttraining compute went to RL.

Other costs sit outside these numbers: labelers (InstructGPT hired 40 contractors), teacher models, RL rewards and environments (chapters 04-05), and research runs, which DeepSeek's figures exclude.

Who uses it and why

"Pre- and mid-training give MAI-Base-1 broad predictive competence and knowledge, but they do not specify how the model should behave, solve long-horizon tasks, or allocate inference-time computation."

"Second, regarding resource allocation, open-source models suffer from insufficient computational investment during the post-training phase, limiting their performance on hard tasks."

"The bootstrap accounts for a small fraction of compute, with the majority being employed for large-scale RL on synthetic and human-created environments."

1. Ouyang et al., Training language models to follow instructions with human feedback (InstructGPT, 2022)
2. Hinton, Vinyals and Dean, Distilling the Knowledge in a Neural Network (2015)
3. DeepSeek-AI, DeepSeek-R1 report (2025), §2-3, App. B.3-B.4, Tables 5 and 7
4. DeepSeek-AI, DeepSeek-V3 Technical Report (2024), Table 1, §5
5. DeepSeek-AI, DeepSeek-V3.2 report (2025), §1, §3
6. DeepSeek-AI, DeepSeek-V4 report (2026), §5.1
7. Microsoft AI, MAI-Thinking-1 report (2026), §3
8. Thinking Machines, Introducing Inkling (2026)
9. Thinking Machines, On-Policy Distillation (2025)
10. Moonshot AI, Introducing Kimi K3.1 Pro (2026)
11. Atlas, posttraining section (coded pipeline grid)

01 Supervised fine-tuning

How does training on example responses teach a base model to answer in a chat format, where did labs get those examples between 2022 and 2026, how many did they use, and what can imitation not teach?

What the loss does

Supervised fine-tuning (SFT) reuses the pretraining objective on a much smaller, curated dataset. Each example is a prompt x and a response y, joined into one token sequence by a chat template (next section). At every position the model predicts the next token, and the loss is cross-entropy: the negative log-probability it gave the correct token. The one change from pretraining is that only response tokens count. The model reads the prompt but is never graded on it.

With easy numbers: take an example of 10 tokens whose last 4 are the response. The model reads all 10, but the loss averages only 4 terms, one per response token, each predicted from everything to its left. If the model gives those 4 tokens probabilities 0.5, 0.5, 0.25 and 0.5, the loss is (0.69 + 0.69 + 1.39 + 0.69) / 4 = 0.87 nats (natural-log units). Over a batch 𝓑 of B examples:

The divisor is a choice:

• Per token, as above: dividing by the batch's response-token count weights every token equally, so long responses count more.
• Per example: averaging within each example first would weight examples equally.

Shapes: tokens [B, L] (L the padded or packed length), logits [B, L, |𝒱|] over a vocabulary 𝒱 of |𝒱| tokens, and a 0/1 loss mask [B, L] marking response tokens.

Masks decide what is learned

Three kinds of mask appear in the reports. The first two act on the loss, the third on attention:

• The prompt. Masking the prompt (loss weight zero) matches how the model is used: at inference the prompt is given, and training on user turns would teach the model to write the user's side of the conversation. The prompt still shapes every gradient, since each response token is predicted from it. The masked share can be large: Xiaomi trained MiMo-V2.6-Distill-Qwen-9B on 77.4B tokens, of which 27.2B were loss tokens, so only 27.2 / 77.4 = 35% of the tokens it read were trained on; the report does not break down the other 50.2B.
• Mistakes inside a response. Z.ai's GLM-5 keeps erroneous steps in agent trajectories as context but masks them out of the loss, "allowing the model to learn error correction behaviors without reinforcing incorrect actions".
• Other examples in the same row. When several short examples are packed into one training row to avoid padding, DeepSeek-V3 adds sample masking so the examples "remain isolated and mutually invisible" (Figure 1.1, bottom). This mask blocks attention between examples, so no token sees a neighbor's text.

Chat templates and special tokens

A chat template turns a list of messages (system, user, assistant, tool) into one token string. It marks roles and boundaries with special tokens, vocabulary entries reserved for the format. DeepSeek-R1's template, stored in its tokenizer_config.json, writes the system prompt after <｜begin▁of▁sentence｜> (the bars are full-width characters), puts <｜User｜> before each user message and <｜Assistant｜> before each reply, and closes each reply with <｜end▁of▁sentence｜>. To request an answer it appends <｜Assistant｜><think> and a newline, so the model starts by reasoning, and it strips the reasoning from earlier assistant turns.

OpenAI's gpt-oss uses a richer format called harmony. Roles are ranked System > Developer > User > Assistant > Tool to resolve conflicting instructions, and "channels" separate analysis (the chain of thought), commentary (tool calls) and final (the answer shown to users). Its tokenizer has 201,088 tokens, extending the o200k tokenizer with format tokens such as <|start|>, <|channel|>, <|message|> and <|end|>.

SFT is where a model learns what these tokens mean, including the end-of-response token that tells it to stop. Training and serving must use the same template: OpenAI says gpt-oss "should only be used with the harmony format as it will not work correctly otherwise". Figure 1.1 puts the template and both kinds of mask on one example.

# Build one example: token ids plus a 0/1 loss mask (simplified R1-style template).
TAGS = {"system": "", "user": "<｜User｜>", "assistant": "<｜Assistant｜>"}

def build_example(tok, messages):
    ids, mask = [tok.bos_id], [0]
    for m in messages:
        head = tok.encode(TAGS[m.role])                 # role tag: never trained
        body = tok.encode(m.content)
        trained = m.role == "assistant"
        if trained:
            body = body + [tok.eos_id]                  # the model must learn to stop
        ids += head + body
        mask += [0] * len(head) + [int(trained)] * len(body)
    return ids, mask

def sft_loss(model, ids, mask):                         # ids, mask: [B, L]
    logp = log_softmax(model(ids[:, :-1]), dim=-1)      # [B, L-1, |𝒱|]
    target = ids[:, 1:]                                 # the next token at each position
    tok_logp = logp.gather(-1, target[..., None])[..., 0]   # [B, L-1]
    m = mask[:, 1:].float()                             # count only response targets
    return -(tok_logp * m).sum() / m.sum()              # mean over response tokens

What imitation teaches, and what it cannot

SFT teaches format, style and instruction following from little data: InstructGPT's first stage used labeler-written demonstrations for about 13k prompts. Its limits come from the loss itself, and there are two:

• It copies the data. SFT minimizes the KL divergence from the demonstrations to the model (chapter 00), so it copies whatever the data contains; Thinking Machines notes that a student "can learn to imitate the teacher's style and confidence but not necessarily its factual accuracy".
• It never trains on its own prefixes. In SFT every prefix y_<t comes from the demonstration; at inference it comes from the model's own samples, so one early mistake puts the model in states it never trained on, and errors compound over long responses. Ranzato et al. (2015) called this exposure bias: a model trained on correct previous tokens is "only exposed to the training data distribution, instead of its own predictions".

RL and on-policy distillation train on the model's own outputs and remove this mismatch (chapters 03 and 07). GLM-5's masked error segments are a partial fix inside SFT: the model reads its mistakes in context, but the loss skips them.

Where the examples come from

The reports name five sources for the responses:

The trend runs from answers people write to answers models write and people or programs check. Data also flows between labs the other way: DeepSeek fine-tuned six Qwen2.5 and Llama models on R1's 800K samples, and Xiaomi fine-tuned Qwen3.5-9B on MiMo-V2.6 outputs. NVIDIA builds much of its SFT data from other labs' models and names the teachers: Llama-Nemotron (March 2025) distilled its reasoning from DeepSeek-R1 by SFT, and for Nemotron 3.5 Super (October 2026) NVIDIA switched its main reasoning teacher from DeepSeek V4 Pro to Kimi K3.

Two recipes turn a model's own samples into SFT data: the cold start, which builds a small, readable seed set for RL, and rejection sampling, which keeps only the samples that pass a check.

Cold-start SFT for reasoning

A cold start is a small SFT stage whose job is to give RL a good starting format. R1-Zero's chains of thought were hard to read and mixed languages, so for R1 DeepSeek built thousands of examples from R1-Zero samples at temperature 1.0:

1. keep those with correct answers and readable format (sympy checks math; rules catch repetition and language mixing);
2. have DeepSeek-V3 refine them and translate the reasoning into the question's language;
3. have human annotators write style examples and verify the results.

The cold start bought instruction following at some cost in reasoning, and the RL after it won most of the reasoning back. DeepSeek's first intermediate checkpoint, Dev1 (the cold-start model), followed instructions better than R1-Zero (on IF-Eval, an instruction-following benchmark, 46.6 to 71.7) but reasoned worse (AIME 2024 77.9% to 59.0%), which DeepSeek attributes to "the limited size of the cold-start dataset". The RL stage after it (giving checkpoint Dev2) brought AIME back to 74.0%.

Rejection sampling: training on filtered own outputs

Rejection sampling builds SFT data from a model's own attempts:

1. sample several responses per prompt;
2. keep those that pass a check, such as a correct final answer;
3. fine-tune on the survivors.

STaR (Zelikman et al., 2022) ran this as a loop: generate rationales with a few worked examples in the prompt, keep those reaching the correct answer, fine-tune, and repeat, restarting each round from the original model "to avoid overfitting".

Why does training on filtered samples help? Because it is a form of RL. Give a reward of 1 for a correct answer and 0 otherwise. RL raises each sampled response's log-probability in proportion to its reward, so correct samples are pushed up and wrong ones, with reward 0, are ignored. Fine-tuning on the correct samples does the same. The STaR paper makes this exact: the expected 0/1 reward is the probability of answering correctly, and the score-function identity (chapter 00) gives its gradient:

The indicator 1(·) zeroes every wrong sample, so this gradient is the SFT gradient on the correct samples. Fine-tuning on filtered samples therefore climbs the probability of answering correctly: it is RL with a 0/1 reward, approximated, because its samples come from a fixed earlier model and are reused for several epochs.

In one careful test, variety mattered more than volume. Yuan et al. (2023), who called the method rejection-sampling fine-tuning (RFT), worked on the GSM8K math word problems. From each SFT model they drew 100 samples per question at temperature 0.7, kept the correct paths, removed duplicates (paths with the same list of equations) and fine-tuned on the original SFT data plus the distinct correct paths:

• LLaMA-7B: of the 100 samples per question, on average 53.3 were correct but only 5.25 distinct; RFT lifted accuracy from 35.9% to 41.7%.
• LLaMA-33B: 88.7 correct paths per question but only 2.78 distinct; RFT did not help it (54.6% to 54.5%).

The gain tracks distinct correct solutions; near-duplicates add little.

Labs run the same step at scale. R1's second SFT took about 600K reasoning samples by rejection sampling from the first RL checkpoint. MAI-Thinking-1's self-distillation fine-tunes on successful RL traces; Microsoft AI found that also including traces with wrong answers "performs similarly", and kept only successful ones because RL produced more than enough.

def rejection_sample(model, prompts, check, k, temperature, distinct):
    data = []
    for x in prompts:
        ys = [sample(model, x, temperature) for _ in range(k)]
        keep = [y for y in ys if check(x, y)]        # correct answer, passes filters
        if distinct:
            keep = dedupe(keep)                      # RFT: one path per equation list
        data += [(x, y) for y in keep]
    return data

def rft(base, sft_data, prompts, check, k=100):
    sft_model = sft(base, sft_data)
    data = rejection_sample(sft_model, prompts, check, k, temperature=0.7, distinct=True)
    return sft(base, sft_data + data)                # RFT keeps the original SFT data too

def star(base, prompts, check, rounds):
    model = base
    for _ in range(rounds):
        data = rejection_sample(model, prompts, check, k=1, temperature=0.0, distinct=False)
        model = sft(base, data)                      # STaR restarts from the base model
    return model

How much is used

Stated SFT sizes use different units (prompts, samples, tokens), so they compare only roughly:

Two findings bear on how much to use:

• More passes over a small set can still help. InstructGPT's SFT models overfit validation loss after one epoch, yet 16 epochs still improved reward-model scores and human preference.
• A large set can hurt the RL that follows. Meta removed more than 50% of Llama 4's SFT data tagged as easy, and 95% for the 2T-parameter Behemoth, because it found that SFT could over-constrain the model and restrict exploration in the RL that followed (chapter 02 quotes Meta's reason). Microsoft AI found that for MAI-Thinking-1 O(1M) traces were enough, and much larger SFT sets risk "over-constraining the policy".

The worked example below sizes one of these sets in tokens and compute.

Who uses it and why

"The motivation is primarily product-driven, with a strong emphasis on enhancing user experience."

"This method ensures that the final training data retains the strengths of DeepSeek-R1 while producing responses that are concise and effective."

"In the expert training stage, the primary role of SFT is to provide a cold start, empowering the model with basic chat, reasoning, and tool-use capabilities, which can then be further enhanced in subsequent expert RL training to achieve improved performance."

1. Ouyang et al., Training language models to follow instructions with human feedback (InstructGPT, 2022), §3
2. Ranzato et al., Sequence Level Training with Recurrent Neural Networks (2015), §1 (exposure bias)
3. Zelikman et al., STaR: Bootstrapping Reasoning With Reasoning (2022), §3
4. Yuan et al., Scaling Relationship on Learning Mathematical Reasoning with Large Language Models (RFT, 2023), §3.3, Table 1
5. DeepSeek-AI, DeepSeek-R1 report (2025), §3-4, App. B.3-B.4 and DeepSeek-R1 tokenizer_config.json (chat template)
6. OpenAI, gpt-oss model card (2025), §2.3, §2.5.1, gpt-oss-120b model page and harmony chat template
7. DeepSeek-AI, DeepSeek-V3 Technical Report (2024), §5.1 and DeepSeek-V3.2 report (2025), §3
8. Z.ai, GLM-4.5 report (2025), §3.1 and GLM-5 report (2026), §3.1
9. Kimi Team, Kimi K2 report (2025), §3.1
10. Xiaomi, MiMo-7B-RL model card, MiMo-V2-Flash report §4.2 and MiMo-V2.6 report §7.1
11. Microsoft AI, MAI-Thinking-1 report (2026), §3.1.4, §3.5
12. Thinking Machines, Introducing Inkling (2026) and On-Policy Distillation (2025)
13. Meta, Llama 4 blog and Llama 4 Scout model card

02 Preference learning and reward models

When no program can check whether an answer is good, how do labs turn judgments like "answer A is better than answer B" into a training signal, and where does that signal still appear in 2025-26 pipelines?

Goals without a checker

A math answer can be checked against the known result, and code can be run against tests. "Be helpful" or "write clearly" has no such check, but people can say which of two answers is better. Christiano et al. found consistent comparisons "much easier" for people than consistent absolute scores, and their simulated robot learned a backflip from 900 comparisons collected in under an hour. Comparisons are still noisy: the trained labelers for InstructGPT, OpenAI's 2022 instruction-following GPT-3, agreed with each other only 72.6% of the time.

From comparisons to a reward model

A reward model (RM) turns comparisons into a score: it reads a prompt x and a response y and outputs one number r(x, y), higher for responses people prefer. It is usually the supervised fine-tuned (SFT) model of chapter 01 with the unembedding replaced by a linear head that maps one hidden state (width d) to one number. With causal attention only the final token's position has seen the whole prompt and response, so the head typically reads the last layer's hidden state there.

The Bradley–Terry model links scores to comparisons: it assumes that the probability people prefer the winner y_w over the loser y_l depends only on the reward gap.

A gap of 0 is a coin flip, a gap of 1.4 gives σ(1.4) = 0.80, and a large gap gives nearly 1. Fitting the RM by maximum likelihood on labeled pairs is logistic regression on reward differences:

Two properties follow:

• Misordered pairs drive learning. The gradient of −log σ(Δ) with respect to the gap Δ is −(1 − σ(Δ)), so confidently ordered pairs barely move the RM and misordered pairs dominate the update.
• Only differences count. A constant added to every reward cancels, so InstructGPT shifted its RM to give human demonstrations a mean score of 0 before RL.

InstructGPT's 6B RMs matched held-out labelers 69.6% of the time, against 72.4% for the labelers they were trained on.

RLHF: maximize the reward, stay near the reference

Reinforcement learning from human feedback (RLHF) then trains the policy π_θ to maximize the RM's score. The RM is accurate only on responses like those it was trained on, so a KL penalty keeps π_θ near a reference policy π_ref, usually the SFT model. This is the KL-regularized objective J(θ) of chapter 00:

InstructGPT optimized this with PPO, an RL algorithm explained in chapter 03, charging the penalty at every token with β = 0.02. Without it, the policy finds answers the RM overrates: Gao et al., with a fixed "gold" RM standing in for people, found the gold score rises and then falls as the policy drifts, while the proxy RM's score keeps climbing. DeepSeek saw this in its reasoning model R1: the helpful RM's reward rose while performance on CodeForces (competitive-programming problems) fell, so R1 used preference rewards only in the last 400 of 1,700 steps of its final RL stage.

DPO: the same objective without a reward model

RLHF needs a reward model and a sampling loop. Direct Preference Optimization (DPO) removes both, starting from one fact: for any reward, the best policy for J(θ) is known in closed form. That policy tilts π_ref toward high reward, multiplying each response's probability by exp(r/β) and renormalizing:

A toy: one prompt has two answers, A and D, which π_ref gives 0.5 each, with rewards r(A) = 1.0 and r(D) = −0.4; take β = 1. The tilted weights are 0.5 · e^1.0 = 1.359 and 0.5 · e^−0.4 = 0.335, so Z = 1.694, π*(A) = 0.8022 and π*(D) = 0.1978. Going backwards, log(0.8022 / 0.5) = 0.473 and log(0.1978 / 0.5) = −0.927: their difference, 1.400, is the reward gap, and each is its reward minus the same constant, log Z = 0.527. In general, taking logs of π* gives

so β times a policy's log-ratio to the reference is the reward it is optimal for, up to a constant per prompt. DPO calls β log(π_θ/π_ref) the implicit reward and puts it into the Bradley–Terry loss in place of r. The constant β log Z(x) appears for both responses of a pair and cancels, so maximum likelihood on the pairs needs only policy log-probabilities:

Instead of fitting a reward model and then solving for its optimal policy, DPO fits the implied reward to the preference data, which fits π_θ directly.

As in the RM loss, each pair's gradient is weighted by how badly the implicit rewards misorder it. Training is one supervised stage with no reward model and no sampling (Figure 2.1), but DPO learns only from fixed pairs and never tries new responses, as online RL (learning from the policy's own fresh samples) does.

import torch, torch.nn.functional as F

def seq_logprob(model, x, y):
    # log π(y|x) summed over response tokens; logits at response positions: [B, L, |𝒱|]
    logits = model(x, y)
    tok = logits.log_softmax(-1).gather(-1, y.unsqueeze(-1)).squeeze(-1)   # [B, L]
    return tok.sum(-1)                                                      # [B]

def reward_model_loss(rm, x, y_w, y_l):
    r_w, r_l = rm(x, y_w), rm(x, y_l)          # one scalar per response: [B]
    return -F.logsigmoid(r_w - r_l).mean()     # Bradley-Terry

def dpo_loss(policy, ref, x, y_w, y_l, beta=0.1):
    with torch.no_grad():                      # π_ref stays frozen
        ref_w, ref_l = seq_logprob(ref, x, y_w), seq_logprob(ref, x, y_l)
    pi_w, pi_l = seq_logprob(policy, x, y_w), seq_logprob(policy, x, y_l)
    margin = beta * ((pi_w - ref_w) - (pi_l - ref_l))   # implicit reward gap
    return -F.logsigmoid(margin).mean()

Harder-to-fool reward models: WARM, WARP and BOND

Google's Gemma 3 bases its RL phase on "improved versions" of three Google DeepMind methods; the report does not say what was improved.

WARM fine-tunes M reward models from one initialization with different hyperparameters and averages their weights, ψ_WARM = (1/M) ∑_m ψ_m, where ψ_m are the weights of RM m. Models fine-tuned from a shared checkpoint tend to be linearly mode connected (the average is at least as accurate as the mean of their accuracies), so WARM gets ensemble-like robustness at one RM's inference cost. A summarization policy trained against WARM beat a single-RM policy 79.4% of the time.

WARP applies the same idea to policies: it averages the weights of several RL-trained policies (weight merging, chapter 07).

BOND distills best-of-N sampling into a single sample. Best-of-N draws N answers from π_ref and keeps the RM's favorite, which costs N generations per answer; BOND trains the policy to produce that distribution in one draw. Let p_≤(y) be the fraction of reference samples that score no better than y. Answer y wins if it is drawn in one of the N slots and the other N − 1 draws score no better, so

The paper's exact form adds a small correction for drawing y more than once. With N = 4, an answer at the 90th percentile of reference scores (p_≤ = 0.9) becomes 4 × 0.9³ = 2.9 times as likely as under π_ref, and a median one (p_≤ = 0.5) half as likely. In logs, log(π_BoN/π_ref) = (N − 1) log p_≤(y) + log N. Matching this to the RLHF optimum π_ref exp(r/β)/Z gives reward r = log p_≤(y) and β = 1/(N − 1), with log N playing the role of −log Z (that is, Z = 1/N). Only ranks matter, so rescaling the RM monotonically changes nothing.

Where preference learning sits in 2025-26

Since R1, reasoning RL in the reports covered here mostly uses checkable rewards (chapter 04). Preference signals remain in narrower roles:

Preferences now arrive late, in small doses, or through an LLM judge reading a rubric. Chapter 04 covers those judges.

"A key learning was that SFT and DPO can over-constrain the model, restricting exploration during the online RL stage and leading to suboptimal accuracy, particularly in reasoning, coding, and math domains."

"We find that more training steps with the model based preference reward signal may lead to reward hacking"

1. Christiano et al., Deep Reinforcement Learning from Human Preferences (2017)
2. Ouyang et al., Training language models to follow instructions with human feedback (InstructGPT, 2022)
3. Gao, Schulman, Hilton, Scaling Laws for Reward Model Overoptimization (2022)
4. Rafailov et al., Direct Preference Optimization (2023)
5. Ramé et al., WARM: On the Benefits of Weight Averaged Reward Models (2024)
6. Ramé et al., WARP: On the Benefits of Weight Averaged Rewarded Policies (2024)
7. Sessa et al., BOND: Aligning LLMs with Best-of-N Distillation (2024)
8. Gemma Team, Gemma 3 Technical Report (2025), §3
9. DeepSeek-AI, DeepSeek-R1 report, §3 and App. B.5
10. Meta, The Llama 4 herd (blog)
11. Z.ai, GLM-4-32B-0414 model card and GLM-Z1-32B-0414 model card
12. Microsoft AI, MAI-Thinking-1 report, §3.4.1
13. Meta, Muse Glimmer report, §5
14. DeepSeek-AI, DeepSeek-V3.2 report, §3 and DeepSeek-V4 report, §5.1.1

03 RL algorithms: from policy gradient to GRPO and back

A model writes eight answers to one math problem and three are graded correct: how do those eight grades become a gradient step, and why did most labs drop PPO's learned critic for GRPO's group average while StepFun kept it and GLM-5.2 brought it back?

The setting

Reinforcement learning (RL) for a language model is a loop: the model writes, a grader scores, and training makes high-scoring responses more likely. The model is the policy π_θ. For a prompt x it samples a response y = (y₁…y_|y|), and a grader returns a scalar reward r(x, y), such as 1 if unit tests pass. Training raises the reward-only objective J_r(θ) = E_{x, y∼πθ}[r(x, y)] (the J(θ) of Chapter 00 also subtracts a KL penalty, which returns below). The reward comes from code or a judge and cannot be backpropagated through.

REINFORCE: scale the log-likelihood gradient by the reward

The simplest method treats each sample as a demonstration weighted by its reward: take the gradient of its log-likelihood, as in supervised fine-tuning, and multiply it by r(x, y). The score-function (log-derivative) identity ∇π = π ∇log π, derived in Chapter 00, shows that this is the gradient of J_r:

This is the REINFORCE algorithm of Williams (1992). Every token of a response shares its reward. For G responses padded to length |y| the log-probabilities form a G×|y| tensor, and the gradient flows only through them.

The estimate is noisy. With rewards in {0, 1}, every correct answer is pushed up equally whether the problem was easy or hard, and wrong answers are never pushed down. One number judges thousands of tokens, and the variance of this estimator "scales unfavorably with the time horizon", which for a language model is the response length.

Baselines and advantages

Subtract a baseline b(x), ideally the prompt's expected reward, and weight each response by its advantage A = r(x, y) − b(x): positive when a sample beat the policy's average, negative otherwise, so wrong answers are now pushed down. A baseline that does not depend on the sampled response keeps the gradient unbiased, because E_y[b(x) ∇log π_θ(y|x)] = b(x) ∇∑_yπ_θ(y|x) = b(x) ∇1 = 0. Weighting by the true advantage (the expected reward after the sampled choice minus the expected reward before it) "yields almost the lowest possible variance".

One update, five choices

Every method in this chapter builds the same update. Token t of response i, one of the responses sampled for a prompt, contributes

REINFORCE with a baseline sets w = 1, N = 1 and no KL term. The methods below differ in five choices about how to weight and limit each token's update, the columns of the comparison table later on:

• Advantage baseline. A learned critic's estimate, or the group's mean reward with or without division by its standard deviation.
• Importance ratio. How a token is reweighted when a batch from an older policy π_old is reused, ρ = π_θ/π_old: per token, per response, or not at all.
• Clip, cap or mask. How far ρ may move before the token's update is cut off or capped.
• Length normalization. Whether N is each response's length, the token count of the group or batch, or a constant.
• KL term. Whether a pull toward a reference policy enters the reward, the loss, or neither.

PPO: reuse a batch, inside a trust region

Proximal Policy Optimization (PPO) answers the five choices with a learned critic, a per-token ratio, a symmetric clip, a token average and, in RLHF-style training, a KL penalty in the reward. The ratio comes from reuse. Sampling is expensive, so one wants several gradient steps per batch, but after the first the samples come from an older policy π_old. Importance sampling corrects for this by weighting each sample with a probability ratio: E_y∼πold[(π_θ(y)/π_old(y)) f(y)] = E_y∼πθ[f(y)]. PPO applies it per token with the importance ratio ρ_t = π_θ(y_t | x, y_<t) / π_old(y_t | x, y_<t) and maximizes the surrogate E[ρ_tA_t]. The per-token ratio is a cheap stand-in for the whole-response ratio, which GSPO revisits below.

Unchecked, ρ_t drifts far from 1; several steps on the plain policy-gradient loss "often leads to destructively large policy updates". PPO therefore clips the ratio to a trust region, a band around the old policy:

The min makes the clip one-sided, and the sign of the advantage picks the side:

• A_t > 0. The min equals min(ρ_t, 1+ε)·A_t: once a good token's probability has grown by the factor 1+ε, the objective is flat and the token gives no more gradient in this batch.
• A_t < 0. The min equals max(ρ_t, 1−ε)·A_t: once a bad token has fallen to 1−ε times its old probability, pushing it lower earns nothing.

In the other two cases (a good token became less likely, or a bad one more likely) the unclipped term is the smaller, so its gradient flows and the policy can undo a harmful move. The result is a pessimistic lower bound on the unclipped objective. Figure 3.1 draws all four cases.

For the baseline, PPO trains a value function V, also called the critic: a second network that predicts the expected future reward from each prefix s_t = (x, y_<t), so every token gets its own advantage. Generalized advantage estimation (GAE) builds that advantage from the critic's one-step errors δ_t = r_t + γV(s_t+1) − V(s_t), with r_t the reward paid at token t and γ a discount factor, summed with a decay λ:

λ trades bias against variance:

• λ = 0. The advantage is the critic's one-step error: low in variance but biased when V is wrong.
• λ = 1. The advantage is the return (the discounted sum of rewards from token t to the end) minus V(s_t): unbiased but noisy. With γ = 1 and a single reward at the end, the return is just r(x, y).

In RLHF-style PPO the KL term lives in the reward. The reward model usually scores only the final token, and every token pays −β log(π_θ/π_ref), which holds the policy near a reference policy π_ref (usually the SFT model) with KL coefficient β.

The critic is what costs. It is "typically another model of comparable size as the policy model", so PPO trains two large networks. DeepSeek's R1 report adds that with only a final reward the critic must predict success from a partial chain of thought that may later be revised, and that PPO is sensitive to λ. On MATH with DeepSeek-Coder-V2-Lite, a 16B-total MoE (mixture-of-experts: a router sends each token to a few of many expert MLPs; pretraining notes, Chapter 02), PPO at the common default λ = 0.95 did considerably worse than GRPO and came close only at λ = 1.0.

GRPO: the group is the baseline

Group Relative Policy Optimization (GRPO), introduced with DeepSeekMath, changes the first choice: the other samples for the same prompt replace the critic as the baseline. For each prompt it samples a group of G responses from π_old (64 in DeepSeekMath, 16 in DeepSeek-R1), grades them, and gives every token of response i one advantage, its reward R_i measured against the group:

Otherwise GRPO keeps PPO's per-token ratio and clip, averages each response's token terms over its length |y_i| before the group, and moves the KL term into the loss, estimated per token as π_ref/π_θ − log(π_ref/π_θ) − 1.

Figure 3.2 shows the group baseline at work on pass/fail rewards.

# One GRPO update. policy and ref return per-token log-probs, shape [|y|].
def grpo_update(policy, ref, prompts, G, eps, beta, n_minibatches, opt):
    batch = []
    for x in prompts:                                   # weights now = pi_old
        ys = [policy.sample(x) for _ in range(G)]
        R = torch.tensor([float(reward(x, y)) for y in ys])  # [G]
        A = (R - R.mean()) / (R.std(unbiased=False) + 1e-6)
        for y, a in zip(ys, A):
            with torch.no_grad():
                batch.append((x, y, a, policy.logprobs(x, y), ref.logprobs(x, y)))
    for mb in split(batch, n_minibatches):              # pi_theta drifts from pi_old
        loss = 0.0
        for x, y, a, logp_old, logp_ref in mb:
            logp = policy.logprobs(x, y)                # [|y|], with grad
            rho = torch.exp(logp - logp_old)
            surr = torch.min(rho * a, torch.clamp(rho, 1 - eps, 1 + eps) * a)
            d = logp_ref - logp                         # log(pi_ref / pi_theta)
            kl = torch.exp(d) - d - 1                   # per-token estimate, >= 0
            loss = loss - (surr - beta * kl).mean() / len(mb)  # mean over tokens, then responses
        opt.zero_grad(); loss.backward(); opt.step()

What broke at scale, and the fixes

On long chains of thought and large MoE models, GRPO's settings were changed one or two at a time. The table gives each method's five choices; the paragraphs say which ones it changed and why.

DAPO (ByteDance Seed, 2025), built while training Qwen2.5-32B base on math, changed the clip and the length normalization, and added two fixes to what the model trains on:

• Clip-higher raises ε_high above ε_low. With ε = 0.2 a token at probability 0.01 can rise only to 0.012 per batch, while one at 0.9 is effectively unbounded (0.9 × 1.2 = 1.08 exceeds 1, so the clip never binds), so rare tokens stay rare and the policy's entropy (the spread of its next-token distribution) collapses.
• Token-level loss divides by ∑|y_i| so tokens in long responses are not down-weighted.
• Dynamic sampling over-samples and drops groups with accuracy 0 or 1, whose advantages are all zero.
• Overlong shaping penalizes long answers gradually once they enter the last 4,096 tokens before the cap; answers that hit the cap get the full penalty.

Together they raised AIME 2024 accuracy from 30 to 50.

Dr. GRPO (Sea AI Lab, 2025) changed the baseline and the length normalization after tracing two biases to GRPO's normalizers. Dividing by |y_i| gives short correct answers larger per-token updates and penalizes long wrong answers less per token, so wrong answers drift longer (the 1/6 against 1/14 of the toy example). Dividing by std weights prompts that are almost always or almost never solved more heavily, as in the 1-of-8 case above. Dr. GRPO deletes both divisions.

CISPO (MiniMax, 2025) turned the clip into a cap, to keep the tokens that clipping silences. Training MiniMax-M1, the lab found that reflective tokens such as "However" and "Wait" were rare under its base model, so their ratios jumped after the first update and the clip dropped them from the later updates on the same batch (M1 runs 16 per batch). CISPO clips the importance weight rather than the token update: each token's A·∇log π_θ is multiplied by min(ρ_t, 1+ε_high), held constant, so a token past the cap still contributes at the capped weight, like token 3 of the toy example. There is no lower cap and no KL term. MiniMax kept CISPO for the M2 series, and Mistral trained Leanstral 1.5 with it.

GSPO (Qwen, 2025) changed the importance ratio from one per token to one per response. It argues that ρ_t, an importance weight from a single sample of each next-token distribution, adds noise that accumulates over long responses and, the authors report, can cause irreversible collapse. It clips one length-normalized ratio per response, s_i = (π_θ(y_i|x)/π_old(y_i|x))^1/|y_i|, a geometric mean of token ratios that stays close to 1, which is why its band is tiny; GSPO's and GRPO's ranges "typically differ in order of magnitude". Chapter 06 returns to its claim that this removes the need for routing replay. Qwen names GSPO for Qwen3-Next and Qwen3-Omni, and its use in the July 2025 Qwen3 updates is inferred from Qwen's GSPO blog. Ant uses GSPO for Ling-2.6-1T's agentic specialists; its own LPO, for Ling-1T, sits between token and sequence ratios with one ratio per sentence.

Kimi's objective changes the KL term: a squared penalty on each token's log-ratio pulls toward π_old, not π_ref. It comes from the k1.5 paper (derivation below) and K2 built on it; K2.5 changed it to a per-token form: for each token, the ratio ρ_t times (r − r̄), with r̄ the group's mean reward, minus ω(log ρ_t)², summed over all tokens and divided by the batch's total token count. Tokens whose log-ratio leaves a fixed interval get zero gradient, whatever the advantage's sign. K3's optimization "follows the algorithm in Kimi K2.5".

MIS-PO (StepFun, 2026) changed the importance ratio and the limit: it keeps a critic and drops the importance weight altogether. In Step 3.5 Flash each token's ratio between the trainer's and the inference engine's probability must lie in [0.5, 2], and each trajectory's geometric-mean ratio in [0.996, 1.001]; tokens and trajectories outside are masked, and MIS-PO "treats retained trajectories as effectively on-policy", so they enter the loss with weight 1. Like IcePop (Chapter 06) it filters on the gap between the two engines, but the kept tokens carry no ratio at all.

MAI-Thinking-1, Microsoft AI's reasoning model, changed the clip in two ways. It widens its upper clip bound when entropy falls below a target of 0.3 and narrows it above, in place of an entropy bonus. It also caps ρ_t at ρ_max on every branch, including the two cases that PPO leaves unclipped (a good token that became less likely, a bad one that became more likely), because those branches caused gradient-norm spikes (quote below).

MiMo-V2.6, Xiaomi's agentic MoE, changed the length normalization and the mask. It averages over all tokens of one prompt's G responses (the loss of Chapter 05), which the report says "prevents response length from growing too quickly". Its mask bounds, set separately for each sign of the advantage, are retuned from entropy during training.

Back to a critic: GLM-5.2

The latest change reverses the first choice. Z.ai's GLM models used group-wise GRPO variants through GLM-5, whose reasoning RL used DAPO-style bounds ε_low = 0.2, ε_high = 0.28 (the β = 2 beside them in GLM's report is the IcePop band of Chapter 06, not a KL weight). GLM-5.2's long agent runs use compaction, rewriting the history mid-episode so the work continues in a shorter context (Chapter 05); in GLM's words, a super-long trajectory "is split by compaction into multiple sub-traces". Rollouts of one prompt then yield different numbers of sub-traces of very different lengths, leaving a group average nothing consistent to compare. GLM-5.2 therefore moved to PPO with a critic that estimates token-level advantages for each rollout alone, trained every sub-trace with a token-level loss, and later named the method SAO. GLM does not report the critic's size, its λ, or its cost.

StepFun never dropped the critic. Its Open-Reasoner-Zero (February 2025), which ran RL straight from Qwen2.5 base models, picked PPO over GRPO for the critic's value estimates, with GAE at λ = γ = 1, the value at which DeepSeek found PPO close to GRPO (above). Step-3 trained with a value network supplying the advantages, Step 3.5 Flash runs MIS-PO with a critic at the same λ = γ = 1, and Step 5 Preview pretrains its critic to reduce the variance of the actor loss in long agentic RL.

Who uses it and why

The table follows the release-by-feature grid behind the cross-lab posttraining section of the atlas.

"Since the value model is usually of similar size as the policy model, it introduces a significant memory and computational overhead."

"We therefore move from group-wise optimization to a critic-based PPO formulation that learns from individual rollouts, relying on a critic to estimate token-level advantages rather than group-relative comparisons."

"In practice, however, we found that these unclipped branches sometimes led to catastrophic gradient-norm spikes."

"We select PPO over GRPO due to its superior value estimation enabled by a learned critic."

1. R. J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning, Machine Learning 8 (1992). Ranzato et al. Sequence Level Training with Recurrent Neural Networks, §3.2.1 (2015).
2. Schulman et al. Proximal Policy Optimization Algorithms, §2-3 (2017).
3. Schulman et al. High-Dimensional Continuous Control Using Generalized Advantage Estimation, §1-3 (2015).
4. Shao et al. DeepSeekMath, §4.1 (GRPO) (2024).
5. Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale, §2.3, §3 (2025).
6. Liu et al. Understanding R1-Zero-Like Training: A Critical Perspective (Dr. GRPO), §3.1-3.2 (2025).
7. Zheng et al. Group Sequence Policy Optimization, §3-5 (2025).
8. Kimi Team. Kimi k1.5, §2.3.2; Kimi K2, §3.2.3; Kimi K2.5, §4.4.2; Kimi K3, §4.1.2.
9. DeepSeek-AI. DeepSeek-R1, §2.1, App. A.3; DeepSeek-V3.2, §3.1.
10. Microsoft AI. MAI-Thinking-1 report, §3.1.
11. Xiaomi. MiMo-V2.6 technical report, §5.1.
12. Z.ai. GLM-5 report, §3.2; GLM-5.2 blog; GLM-5.3 blog.

Lab RL objective playground

How does a group of rewards become per-sample advantages, and when does PPO-style clipping stop a token's gradient?

Two small experiments that go with chapter 03. In the first, mark which of eight sampled answers to one prompt were correct and see the advantage each sample gets. In the second, drag the probability ratio of one token and see the clipped objective that PPO and GRPO optimize.

04 Reward design and reward hacking

What produces the number that RL maximizes for a math answer, a code patch or an essay, and what happens when the model finds a way to raise that number without doing the task?

A reward turns a finished response, or the final state of an agent's sandbox, into one number. That number comes from a program that checks the output or from a model that judges it. This chapter takes the two sources in turn, then how labs combine several rewards into one, then what goes wrong when the policy raises the number without doing the task.

Rewards a program can check

When a task has a known answer or a test, a program can compute the reward. The reports describe three kinds of check, the three left columns of Figure 4.1:

• Answer match. DeepSeek-R1 trained reasoning with rule-based rewards. An accuracy reward checks the final answer: math answers go in a fixed format, such as a box, so a rule can compare them with the reference, and code is run against test cases. A format reward checks that reasoning sits between <think> tags. The two are added with equal weight, r_rule = r_acc + r_format; R1 also adds a language-consistency reward (the share of target-language words in the reasoning). Xiaomi's MiMo-7B kept only the correctness checks, with no format or length reward.
• Tests. For software tasks the check is a test suite. MAI-Thinking-1, Microsoft AI's reasoning model, builds tasks from GitHub pull requests (PRs): it runs the tests on the old code with only the PR's new tests added, then with the PR's fix too. Tests that flip from failing to passing (fail-to-pass, F2P) show the issue is solved; tests that pass both times (pass-to-pass, P2P) catch regressions.
• Measured quality. Some tasks grade how good a working result is: Kimi K3's GPU-kernel tasks give zero reward above an error threshold, 0.5 for matching an expert kernel, and up to 1 near the hardware's speed limit.

Rewards a model judges

Open-ended tasks have no answer key, so a model reads the response and judges it. A generative reward model (GRM) is an LLM that reads the prompt, the response and often a rubric (criteria written for that prompt) and writes its judgment as text. The labs differ in who judges and what the judge checks:

• Rubrics. DeepSeek-V3.2 used a GRM with per-prompt rubrics. For instruction following, MAI-Thinking-1 has an LLM judge grade atomic rubric items pass/fail, averaging several passes.
• Pairs. Kimi's judges compare candidates in pairs against rubrics.
• The policy as judge. In DeepSeek-V4 the policy itself is the GRM, and RL trains its judging alongside its answering. In Kimi K2's self-critique rubric reward, K2 judges its own responses in pairs against rubrics.
• Separate graders. Thinking Machines trained Inkling, its open-weights model, with a rubric grader that scores against a checklist and a claims grader that checks each factual claim with agentic web search.

Combining rewards into one number

One response often earns several rewards, and RL needs one advantage. For helpfulness and safety, MAI-Thinking-1 ranks its rewards by priority and combines them in two ways:

• Tie-breaker. In some contexts a lower-priority reward only breaks ties within a group of G samples for the same prompt: if every sample in the group is equally good on the main reward, the secondary reward decides their advantages (chapter 03); if they differ on the main reward, the secondary one is ignored. Main rewards of (1, 1, 1, 1) let the secondary scores set the advantages; main rewards of (1, 0, 1, 0) leave them unused.
• Gate. In other contexts a reward acts as a gate: an unsafe response gets the minimum reward whatever its quality.

Most recipes add one more term, a price on length, which has its own section below.

Figure 4.1 places all of these reward sources on one scale, from fully checkable to fully judged. Its lower two rows preview the rest of the chapter: how each source gets hacked, and what labs do about it.

Length penalties

Outcome rewards say nothing about length, and RL tends to grow it. R1-Zero's thinking lengthened under outcome rewards alone (chapter 08), Kimi k1.5 saw "an overthinking phenomenon" during RL, and learned judges drift toward verbosity (Kimi K3, next section). A long response is paid for twice: every RL step has to generate it, and every user waits for it. A length penalty adds a term that depends on the response's length ℓ in tokens, often counting only the reasoning tokens:

The designs differ in what ℓ is measured against (Figure 4.2):

• Against a hard cap. Generation stops at a maximum length, and a truncated answer usually fails. DAPO's soft overlong punishment ramps the term from 0 to −1 over a band before the cap (a 16,384-token target plus a 4,096-token band; chapter 03): an answer of 18,432 tokens gets (16,384 − 18,432) / 4,096 = −0.5, and one that hits the cap gets −1. Magistral uses the same ramp scaled to −0.1, "to signal the model that the hard cutoff on maximal completion length is near". Kimi K2 truncates and penalizes responses over a per-task token budget, and DeepSeek-V3.2 adds a length penalty without giving its form.
• Against the rest of the group. Kimi k1.5 compares the samples for one prompt: λ = 0.5 − (ℓ − ℓ_min) / (ℓ_max − ℓ_min), so the shortest gets +0.5 and the longest −0.5. Correct answers receive λ and wrong ones only min(0, λ): a long wrong answer is pushed down, but a short wrong one gains nothing. Three correct answers of 2,000, 4,000 and 6,000 tokens get +0.5, 0 and −0.5. Ling-2.6-1T reuses this form with difficulty-specific length limits. NVIDIA's Nemotron 3 Nano adds a zero-mean, group-relative length bonus in RLHF, plus 0.5 for the shortest response only if its base reward reaches 80, so that conciseness bonuses go only to high-quality responses. MiMo-V2.6 penalizes only correct rollouts longer than a quantile of the correct lengths in their group, and only when the group's pass rate clears a threshold, which "preserves room for exploration on difficult prompts".
• Against a reference model. Kimi K3 sets a per-problem budget from its cold-start model (the model after initial SFT): a candidate longer than a set multiple of that model's typical length automatically loses its comparison (chapter 08 gives the numbers).
• Inside the judge. When a generative reward model scores helpfulness, length can be priced there. For Nemotron 3.5 Super, NVIDIA trained a variant of its Nemotron 3 Ultra GenRM with a reasoning length penalty.
• Redundancy instead of length. Ant argues that token length alone cannot stop repeated reflection. Ling-2.5-1T's reward combines "Correctness" and "Process Redundancy", and Ling-2.6-1T adds an LLM-judged semantic redundancy penalty and, for agents, a repetition penalty from the zlib compression ratio: repetitive text compresses well, so it is penalized harder.
• Time instead of tokens. MiniMax scores M2.5's agent trajectories by task completion time, trading capability against speed.
• Set by the request. Effort levels make the price of length depend on the level the user asks for: DeepSeek-V4's per-mode specialists, DeepSeek-V4.1's numeric effort, and Ring-2.6-1T's moderate penalties for its high mode and minimal ones for xhigh (chapter 08).

Penalties have failure modes of their own. A plain penalty can be hacked by collapsing into overly terse answers, so MAI penalizes only lengths outside a per-context range. Applied from the first step, a penalty can slow training early on, so Kimi k1.5 starts without it and adds a constant one later. And length can fall with no penalty at all: Inkling's chain of thought grew terser during RL, dropping grammatical overhead, and Thinking Machines says "efficiency alone drove the compression".

A length penalty changes the reward. Length normalization in the loss (chapter 03) is a different lever: it sets how much each token of a long response counts in the update. It can also hold length back, as MiMo-V2.6's group-level token average is meant to, but it never scores a response lower for being long.

Reward hacking

Reward hacking is earning reward in a way the designer did not intend, with solutions "valid in some literal sense but don't meet the designer's intent" (Amodei et al.). The route depends on the kind of reward:

• Learned judges are hacked through their biases: R1's helpful RM (chapter 02), rubric graders rewarding lists of plausible facts (Inkling), drift toward verbosity (Kimi K3).
• Bonus terms are hacked by maximizing them for their own sake. Kimi K2.5 gives an orchestrator model a bonus for starting subagents (helper copies that run sub-tasks in parallel), and it learned to spawn many subagents without meaningful task decomposition.
• Checkable rewards are hacked through the environment around the check. Instead of solving the task, the model finds the answer elsewhere or changes what the check sees. In repository repair, Xiaomi calls the recurring case solution leakage: the agent obtains a published fix from outside its task and builds its patch from it. Reported cases:

Hacking tends to grow with capability: Moonshot says stronger agents "explore more aggressively and may even attempt reward hacking".

Defenses

The reported defenses act at five points, from mid-training to auditing the run:

• Teach the model before RL. MiMo turns hacking cases from early experiments into mid-training examples (mid-training is the stage between pretraining and posttraining). In each, the model reflects on the faulty reasoning, revises the hacking turn and continues from the task specification, with the original error left recognizable; Xiaomi reports that the examples improved alignment.
• Clean the environment. Building a task with its reference patch applied can leave the answer behind, so MiMo removes build logs, verifier outputs, residual patches and generated binaries or bytecode, and cleans caches, including those outside the repository, while keeping the third-party dependencies needed to rebuild offline. MAI and MiMo delete git commits after the task's base commit, and both cut or restrict network access; MiMo also gives explicit instructions against retrieving existing solutions. V4.1-Flash adds per-sandbox kernel security rules for files and network.
• Harden the check. MAI resets every test file the agent changed and applies the new tests only at grading. GLM-5.3 writes verifiers without the reference solution, closes shortcuts that solver trajectories reveal, and treats a verifier as reliable enough to train on once it passes "oracle, no-op, and unsolved-state checks". The blog does not define these; the names suggest that the reference solution must pass, doing nothing must fail, and the unsolved starting state must fail. Kimi K3 pairs public verifiers that give feedback with hidden ones on held-out cases.
• Catch hacks during the rollout. A rollout is one sampled attempt at a task, including every tool call. GLM-5.2 screens each tool call in two stages: a rule filter for recall, then an LLM judge of intent for precision. A blocked call returns dummy output and the rollout continues, which Z.ai says helps prevent the training instability and model collapse that stopping rollouts abruptly can cause.
• Search and audit. Before training, MiMo sends a dedicated hack agent after its prepared environments, guided by exploits from early runs such as solutions recovered from cached artifacts or preinstalled copies of the project; each round of findings refines the cleanup, and the agent runs again until it finds no exploit in any environment. During training the policy may find shortcuts the screening missed, so MiMo audits trajectories offline and its grader sets the reward of confirmed hacks to zero before computing group statistics; their share stayed below 2%. MAI has an LLM monitor flag rollouts for human review.

def coding_rollout(policy, env, task, rule_filter, judge, max_turns):
    """Sketch: GLM-5.2-style online hack guard, then MAI-style grading."""
    history = [task.statement]
    for _ in range(max_turns):
        call = policy.next_tool_call(history)            # e.g. a shell command
        if call.is_submit:
            break
        if rule_filter(call) and judge.is_hack(call, history):   # recall, then precision
            result = "[blocked]"                         # dummy output, rollout goes on
        else:
            result = env.run(call)
        history += [call, result]
    env.restore(task.test_files)                         # undo any edits to tests
    env.apply(task.hidden_tests)                         # tests the agent never saw
    passed = env.run_tests(task.fail_to_pass + task.pass_to_pass)
    return 1.0 if all(passed) else 0.0                   # binary reward

Rewarding honest confidence

A plain correctness reward teaches guessing, since any answer with a chance of being right beats "I don't know". Thinking Machines trained Inkling for calibration "with RL against proper scoring rules on a large corpus of resolved real-world questions." A scoring rule grades a reported probability q once the outcome o ∈ {0, 1} is known (o = 1 if the event happened). It is proper if reporting one's true belief p gives the best expected score, and strictly proper if that optimum is unique (Gneiting and Raftery). Three common rules:

• Brier loss (q − o)², a loss where lower is better (as an RL reward, use −(q − o)²): strictly proper.
• Log score, log q if o = 1 and log(1 − q) if not, a reward where higher is better: strictly proper.
• Linear score, a reward equal to the probability given to the actual outcome: not proper.

Rubric rewards can push the other way. Kimi K2's report says its rubrics penalize hedging such as "I might be wrong", which may favor confident answers on ambiguous questions. Rewards for saying "I don't know", including Inkling's and MAI-Thinking-1's, are covered in chapter 10.

"We find that GLM-5.2 shows more potential hacking behavior than GLM-5.1. This makes the verification signal easy to optimize, but fails to actually improve the fundamental capabilities of the model."

"Rubrics can penalize errors in principle, but in practice they emphasize recall and can be hacked by models spraying plausibly relevant facts hoping to match rubric items."

1. DeepSeek-AI, DeepSeek-R1 report, §2.2, §3.2.1, App. B.5
2. Xiaomi, MiMo-7B report, §3.2
3. DeepSeek-AI, DeepSeek-V3.2 report, §3, §3.2.3; DeepSeek-V4 report, §5.1.1; DeepSeek-V4.1-Flash report, §5.1.3-5.1.4
4. Kimi Team, Kimi K2 report, §3.2.2-3.2.3, App. F; Kimi K2.5 report, §3; Kimi K3 report, §4.1.2, §4.2.4-4.2.7
5. Z.ai, GLM-5 report, §3.4; GLM-5.2 blog; GLM-5.3 blog
6. Microsoft AI, MAI-Thinking-1 report, §3.3.1, §3.4
7. Xiaomi, MiMo-V2.6 report, §4.2.6
8. Thinking Machines, Introducing Inkling (Epistemics)
9. Amodei et al., Concrete Problems in AI Safety (2016), §4
10. Gneiting and Raftery, Strictly Proper Scoring Rules, Prediction, and Estimation (JASA 2007)

05 Environments and agentic RL

What does an environment for training an agent contain, and how did labs scale from grading one written answer to grading multi-turn agents in hundreds of thousands of sandboxed tasks?

What an environment is

Chapter 03's RL loop needs a prompt x, a sampled response y and a reward r(x, y). DeepSeek-R1, DeepSeek's January 2025 reasoning model, trained on 26K math questions, 17K algorithm problems and 8K bug-fixing problems. A bug-fixing prompt holds a GitHub issue, the buggy source and failing unit tests; the reward checks whether the corrected code passes. The report describes no tool use, so the checker is the whole environment.

An agent also needs a world to act on. DeepSeek-V3.2 calls this an "<environment, tools, task, verifier>" tuple. In the terms an episode uses, an environment has four parts:

• Task: an instruction plus an initial state, such as a repository at one commit.
• Actions: tool calls such as shell commands, file edits and web searches.
• Observations: the output returned after each call.
• Verifier: the check that turns the final state into a reward.

One episode

ReAct (Yao et al., 2022) formalized the loop: at turn k the agent sees the context c_k = (o₁, a₁, …, a_k−1, o_k) and picks action a_k (k counts turns). A harness or scaffold, the program around the model, runs each tool call in a sandbox (an isolated container or virtual machine) and appends the output. MAI-Thinking-1, Microsoft AI's reasoning model, ends the loop, and with it the rollout (one sampled episode), "when the model emits no tool calls, or when the rollout exceeds the step, context-length, or time budget". Figure 5.1 follows one episode of a software task, and shows what the trainer receives from it.

For training, the episode is one token sequence; in agentic RL the "response" y_i is this whole trajectory. A mask m_i,t ∈ {0, 1} marks the tokens the policy generated. MiMo-V2.6, Xiaomi's September 2026 release, writes its loss in this form, chapter 03's update with the mask added and the per-token importance ratio omitted here:

Here y_i and m_i have |y_i| entries and A_i is one scalar per trajectory. Tool outputs sit in y_i,<t and condition later tokens, but m_i,t = 0 keeps them out of the gradient: a policy gradient reweights the policy's own choices, and the policy did not choose the tool output. One reward covers every policy token; MAI applies credit "uniformly across all tokens from all policy steps".

def run_episode(policy, sandbox, max_turns, max_tokens):
    x = sandbox.task_prompt()                # task x
    y, mask = [], []                         # trajectory y and its loss mask m
    for k in range(max_turns):
        out = policy.generate(x + y)         # one turn: reasoning + optional tool call
        y += out; mask += [1] * len(out)
        call = parse_tool_call(out)
        if call is None:                     # no tool call: the agent has finished
            break
        obs = sandbox.execute(call)          # e.g. run a bash command
        y += obs; mask += [0] * len(obs)
        if len(x) + len(y) > max_tokens:     # context budget exhausted
            break
    return x, y, mask, sandbox.verify()      # e.g. run hidden F2P and P2P tests

def agentic_loss(policy, episodes):          # G episodes of the same task
    r = [reward for _, _, _, reward in episodes]
    A = [(ri - mean(r)) / (std(r) + 1e-6) for ri in r]   # group advantage, chapter 03
    total = sum(len(y) for _, y, _, _ in episodes)        # sum of |y_i|, tool tokens included
    loss = 0.0
    for i, (x, y, mask, _) in enumerate(episodes):
        logp = policy.logprobs(x, y)         # log π(y_t | x, y_<t) for each t
        loss -= A[i] * sum(m * lp for m, lp in zip(mask, logp))
    return loss / total

How labs build environments at scale

The pipelines in these reports share one shape:

1. start from real material, such as merged pull requests or Stack Overflow posts, or synthesize it, such as search questions about long-tail entities or a closed world with a seeded database;
2. have an LLM agent build the executable part: a container image, a tool set, a test script;
3. validate each candidate, for example by requiring that the reference solution pass and the unsolved state fail;
4. keep the survivors, often a small fraction of the candidates.

Software engineering: repositories become environments

Software engineering is the most developed case, and SWE-bench (Jimenez et al., 2023) set its template. From about 90,000 pull requests (PRs) in 12 Python repositories it kept merged PRs that resolve an issue and add tests, and required a fail-to-pass (F2P) test, one that fails before the fix and passes after; 2,294 tasks remained. MAI also checks pass-to-pass (P2P) tests, which must keep passing.

Labs scaled this up by having an LLM agent build each repository's container image (a snapshot of the code with its dependencies installed). DeepSeek-V3.2 accepted an environment only if the reference patch gave a nonzero F2P count and zero pass-to-fail tests, and trained on 24,667 code-agent tasks. MiMo-V2-Flash, Xiaomi's 309B-total mixture-of-experts model (MoE: each token passes through only a few of many expert sub-networks; pretraining chapter 02), reports 70% build success on over 10,000 concurrent containers (Kubernetes pods) and RL across approximately 120K environments; GLM-5 (Z.ai) built over 10K environments in 9 languages. Qwen3-Coder-Next (February 2026) reports 807,693 task instances from real pull requests in 52,960 repositories, plus about 800K with injected bugs, each in an executable Docker environment.

Terminal, search and general tool use

Terminal. Terminal tasks set a goal reached through a shell and checked by a test script. GLM-5 turned LLM-drafted tasks into Harbor-format tasks (a description, a Docker environment and tests), giving thousands of environments with over 90% of Docker builds succeeding. MiMo-V2-Flash built about 30,000 from Stack Overflow and Stack Exchange material.

Search. Search questions must be hard to answer and easy to check. DeepSeek-V3.2 had an agent build questions about long-tail entities with web search, and kept one only when a verification agent confirmed its answer and rejected every other agent's candidate; with filtered prompts from its existing helpful-RL data added, the set had 50,275 search tasks.

General tool use (booking, inventory, support) runs in synthesized closed worlds: a seeded database, tool functions, tasks and a checker. DeepSeek-V3.2's synthesis agent wrote tools, a task, a solution and a verifier, then raised the difficulty, keeping tasks with nonzero pass@100 (solved at least once in 100 samples): 1,827 environments and 4,417 tasks. MAI reports more than 150 environments and 130,000 tasks, often with over 50 tools each. GLM-5.3's research agents turn real work into long-horizon environments, some worth days of an engineer's time, and each synthesized verifier is tested before its binary reward is used for training.

Stated task or environment counts (blank: not reported):

MiMo-V2-Flash's task table has no terminal row; its 30K "code agent" tasks with synthesized prompts are most likely the ~30K terminal tasks built from Stack Overflow and Stack Exchange, so they are counted once, under terminal.

Training across harnesses

The same task can reach the model through different harnesses. The harness sets the system prompt, which tools exist and how their calls are written, what tool responses look like, and how the context is shortened when it fills. A model that trains inside only one harness can learn the harness along with the task: Xiaomi warns that training in a single harness may couple task-solving strategies to that harness, and users run models in harnesses the lab never trained in.

A toy case: harness A edits files with edit_file(path, old, new) and summarizes the history at 80% of the window; harness B accepts only unified diffs through apply_patch and never summarizes. A policy trained only in B can learn to answer every edit with a diff, which A cannot apply, although the bug and its fix are the same in both.

Labs vary the harness in three ways:

• Train in real harnesses. DeepSeek-V4.1 trained jointly across Claude Code versions, and in another run across OpenCode, Pi and its own harness. Kimi K3, Moonshot's 2.8T-total MoE, assembles harnesses from modules that can reproduce Kimi Code, Claude Code, Codex, OpenClaw and Hermes (agent harnesses from different developers), and gives different task groups different harness configurations during RL.
• Train in small harnesses built for RL. Xiaomi argues that production harnesses fit RL poorly, for two reasons. Their safeguards, workflows and instruction prompts fall outside the task-completion reward signal, making credit assignment unreliable, and their modules are tightly coupled rather than independently configurable, so one mechanism cannot be varied on its own. MiMo-V2.6 instead starts every mini-harness from the same minimal agent loop (a system prompt, tools and context management), keeps those modules decoupled, and recombines them into configurations for Code, General, Visual and Cyber tasks.
• Perturb the interface. Inkling, Thinking Machines' open model, randomized tool sets and schemas "to reduce sensitivity to any particular one". MiniMax built M2's agent data (October 2025) for robustness to "perturbations across the model's entire operational space", including tools, system prompts and tool responses, and Qwen3-Coder-Next trains on 21 tool-call templates, finding that more templates make it more robust to unfamiliar formats.

Mixing harnesses also changes the RL system. In MiMo's large batches each group of samples is bound to one harness, so a Harness Pool keeps many harness instances running at once, each configured independently (chapter 06 covers the rest of the RL system).

What goes wrong when episodes get long and many

Besides reward hacking through the environment (chapter 04), long and numerous episodes raise two problems; which harness to train in is the previous section.

The context overflows. DeepSeek-V3.2, with a 128K window, tested three rules at evaluation, triggered at 80% of the window: summarize and restart, discard the oldest 75% of tool history, or discard all of it. Compaction is such a rewrite done by the harness inside an episode: when the context fills, it shortens the history (for example into a summary) and the episode continues in a fresh context, so one long trajectory becomes several sub-traces.

Sub-traces are awkward for group-relative advantages, which compare G whole responses: GLM-5.2 found that "different rollouts under the same prompt yield different numbers of trainable traces with highly variable lengths". It trained every sub-trace with a token-level loss and returned to a critic for advantages; chapter 03 explains that switch.

The sandboxes must scale and contain the agent. Kimi K3 created 51,219,741 sandboxes from 1,505,678 images in training and evaluation, added Firecracker microVMs (small, fast-starting virtual machines) after agents crashed the operating-system kernel in container sandboxes, and pauses a sandbox while it waits for the model, a wait that can fill 98% of its lifetime. DeepSeek-V4.1's training needed "millions of concurrent sandbox instances" and packs over 2,500 live containers per node, up from about 1,000.

Who uses it and why

"large-scale RL on synthetic data yields substantial improvements over DeepSeek-V3.2-SFT on Tau2Bench, MCP-Mark, and MCP-Universe benchmarks. In contrast, restricting RL to code and search scenarios does not improve performance on these benchmarks, further highlighting the potential of synthetic data."

"Training with a single fixed agent harness can cause a model to overfit to a particular tool schema, system prompt, context management mechanism, or interaction protocol."

"As agent capability improves, much of the difficulty in scaling post-training moves from the model to the environment."

1. Yao et al., ReAct: Synergizing Reasoning and Acting in Language Models (2022), §2
2. Jimenez et al., SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (2023), §2.1
3. DeepSeek-AI, DeepSeek-R1 (2025), App. B.3.1 Table 4
4. DeepSeek-AI, DeepSeek-V3.2 (2025), §3.2.3 Table 1, §4.3, §4.4
5. DeepSeek-AI, DeepSeek-V4.1 Technical Report (2026), §5.1
6. Microsoft AI, MAI-Thinking-1 report (2026), §3.3
7. Xiaomi, MiMo-V2-Flash Technical Report (2025), §4.3
8. Xiaomi, MiMo-V2.6 Technical Report (2026), §4.1
9. Z.ai, GLM-5 Technical Report (2026), §4.2
10. Z.ai, GLM-5.2 blog (2026), RL for Long-Horizon Task with Anti-hacking
11. Z.ai, GLM-5.3 blog (2026), Stronger Coding
12. Moonshot AI, Kimi K3 Technical Report (2026), §4.2.1, §5.3.2
13. Thinking Machines, Introducing Inkling (2026)

06 RL systems: keeping rollouts and training in step

When an agent rollout can run for hundreds of tool calls on an inference engine separate from the trainer, how do labs keep each sample close enough to the weights being trained that the importance ratios of Chapter 03 still mean something?

One RL step

Each RL step in Chapter 03 runs four jobs, usually split between two systems (Figure 6.1):

1. Generate. A rollout engine, an inference server such as SGLang or vLLM, samples responses and runs tools, and can log π_infer(y_t), the probability it gave each sampled token.
2. Score. A grader scores each response.
3. Train. A trainer, such as Megatron-LM, recomputes the probabilities of the sampled tokens, computes the loss and updates the weights.
4. Sync. The new weights are copied back to the rollout engine.

Rollout is a large share of the bill: MiMo-V2.6-Pro's RL cost split 43.8% rollout, 43.5% training and 12.7% grading, and in MAI's largest job 4,096 of 4,864 GB300 chips ran inference.

Two separate things can go wrong between steps 1 and 3:

• Staleness is about when. If the steps overlap, the engine sampled with weights several updates older than those being trained.
• Training–inference mismatch is about where. Even with identical weights and no staleness, engine and trainer run different code and compute slightly different probabilities.

The importance ratio separates them into two factors. Let π_θ be the trainer's probability under the weights being updated and π_train its probability under the weights that sampled the token (Chapter 03's π_old, computed in the trainer):

Toy numbers: the engine logged 0.30 for a token; the trainer computes 0.31 with the same weights and 0.36 with its current ones. The ratio 0.36 / 0.30 = 1.20 is 0.36 / 0.31 ≈ 1.16 from the policy moving times 0.31 / 0.30 ≈ 1.03 from the systems disagreeing.

Synchronous RL and the long tail

In synchronous RL the four steps run in order: the engine samples a whole batch with the current weights, the trainer updates, and the next batch waits for the new weights (Figure 6.2, top). Kimi runs this way with both engines colocated on the same GPUs, each offloading while the other works, and K2 copies a full parameter update to the engines in less than 30 seconds.

The cost is the long tail. A batch is done only when its longest rollout is, and Kimi describes K3's agent environments as running "often over hundreds or thousands of tool calls". GLM-5 gives the idle GPUs this causes as its reason for making agentic RL asynchronous (quote below). Even a synchronous step is slightly off-policy once its batch is split into several mini-batch updates (worked example below).

Asynchronous RL and how labs bound staleness

Asynchronous RL (Figure 6.2, bottom) lets the steps overlap: rollout workers keep generating with whatever weights they hold, and the trainer updates whenever enough finished samples have accumulated. Staleness is the number of policy updates between the weights that generated a sample and the weights being trained. AReaL, an open asynchronous RL system from Tsinghua and Ant Group, caps the staleness allowed in each batch (a cap of 0 is synchronous RL) and reports up to 2.77× faster training than synchronous systems on the same GPUs.

Labs limit staleness in three ways, often combined:

• Cap the age. MAI-Thinking-1, Microsoft AI's reasoning model, takes 5 gradient steps between weight updates of the inference engines and discards rollouts more than 8 updates stale. MiMo-V2.6, Xiaomi's agentic MoE, runs GRPO "with asynchronous partial rollouts at a staleness of 4". NVIDIA's Nemotron 3 Super runs GRPO asynchronously on separate training and inference GPUs and keeps the inference workers at most one update behind. GLM-5 runs agentic RL asynchronously on separate GPUs and drops a sample whose oldest tokens come from a version too far behind.
• Bound or mask per token. DeepSeek-V4.1-Flash made nearly all RL and on-policy distillation asynchronous while keeping rollout and training on the same GPUs, where training preempts unfinished rollouts. It bounds the maximum off-policy ratio, masks the loss on tokens with excessive staleness, and discards early-returned short samples, because short responses finish first and would skew the first batches.
• Adjust the optimizer. MAI lowered its learning rate at long generation lengths because the longest rollouts are the most off-policy. GLM-5 resets the optimizer after each weight push, because each push "considers a different optimization problem".

Earlier, MiMo-7B had avoided asynchronous training because it "introduces staleness in long-sequence responses".

Partial rollouts: new weights in the middle of a rollout

A long rollout may still be running when new weights arrive. Labs handle it in four ways:

• Pause and resume next iteration. Kimi's k1.5 paper introduced partial rollouts: if a trajectory exceeds the iteration's token budget, the unfinished part is saved and continued in the next iteration. K3 pauses generation once a set fraction of the iteration's trajectories has finished and resumes the paused ones first in the next iteration. Kimi calls this RL synchronous, but a long trajectory now holds tokens from several policy versions.
• Keep the cache. DeepSeek-V4.1-Flash can stop a rollout at any token and keeps its KV cache (the stored attention keys and values of earlier tokens; pretraining notes, Chapter 03) and expert routing, so it resumes on the new checkpoint without recomputing the prefix.
• Rebuild the cache. MiMo-V2.6 instead re-runs prefill, the pass that rebuilds the KV cache for the whole prefix (pretraining notes, Chapter 00), after each update, and uses a large batch to amortize that cost.
• Swap weights in flight. Mistral's Magistral (June 2025) and Nemotron 3 Super never pause: new weights reach the engines mid-generation (Magistral broadcasts them in under 5 seconds), and in-flight rollouts continue on the KV cache computed with the old weights, as DeepSeek-V4.1-Flash's resumed rollouts do.

Correcting for staleness in the loss

A trajectory from several versions breaks the assumption that one π_old produced every token. The common fix takes the ratio's denominator from the rollout engine itself: π_infer, logged at sampling time by whichever version produced the token. This ratio contains both factors of the split above, and labs keep it in check in three ways:

• Mask tokens outside a band. GLM-5's agentic RL puts π_infer in place of π_old in the ratio and masks tokens whose ratio π_θ/π_infer leaves [1−ε_low, 1+ε_high]. MiMo-V2.6 also divides by the rollout probability and does not recompute it for partial rollouts.
• Split the ratio. AReaL writes π_θ/π_behav = (π_prox/π_behav)·(π_θ/π_prox), where π_behav is the policy that sampled the token and π_prox a recent one. The first factor is a fixed weight and only the second is clipped to [1−ε, 1+ε], so old samples do not pull the update back toward old weights. With ε = 0.2 and a token at 0.20, 0.25 and 0.27 under the old, recent and current policies, plain clipping sees 0.27 / 0.20 = 1.35, outside [0.8, 1.2]; AReaL clips only 0.27 / 0.25 = 1.08 and keeps 1.25 as a fixed weight.
• Tolerate it through the regularizer. Kimi relies on its squared log-ratio penalty, which K3 says "inherently tolerates such an extreme off-policy regime".

# Trainer side of asynchronous RL (sketch). Rollout workers run in parallel and
# put finished samples on a queue. Each sample s carries s.version (the weights it
# started with), s.logp_infer (per-token log-probs the engine logged, [|y|]) and
# s.A (its group-relative advantage, Chapter 03).
def trainer_loop(policy, queue, engines, B, max_stale, eps_low, eps_high, opt):
    version = 0
    while True:
        batch = []
        while len(batch) < B:
            s = queue.get()                          # waits for a finished rollout
            if version - s.version <= max_stale:     # drop samples that are too stale
                batch.append(s)
        loss = 0.0
        for s in batch:
            logp = policy.logprobs(s.x, s.y)         # [|y|], with grad
            rho = torch.exp(logp - s.logp_infer)     # pi_theta / pi_infer, per token
            keep = (rho > 1 - eps_low) & (rho < 1 + eps_high)   # mask, not clip
            loss = loss - (keep * rho * s.A).mean() / B
        opt.zero_grad(); loss.backward(); opt.step()
        version += 1
        for e in engines:                            # in-flight rollouts continue
            e.load_weights(policy.state_dict(), version)   # on the new weights

Training–inference mismatch: same weights, different numbers

Logging π_infer exposes the second factor. With identical weights, the trainer's π_train(y_t) and the engine's π_infer(y_t) still differ, because the two use different kernels, reduction orders, batch shapes and precision. In a mixture-of-experts (MoE) model, where a router sends each token to a few experts (pretraining notes, Chapter 02), a rounding error can change which experts are picked. Illustratively, scores of 0.3001 and 0.2999 for a token's second and third expert swap under a 0.0003 error. Ma et al. measured Qwen3-30B-A3B rolled out in SGLang and replayed in Megatron: about 10% of router decisions differed, and 94% of tokens used a different expert in at least one layer. The ratio ρ_t then mixes real policy change with engine noise, and the Ring-1T paper argues that the gap can grow from one training step to the next.

How labs shrink the mismatch

The fixes work at three levels: replay the engine's discrete choices in the trainer, make both systems compute the same numbers, or drop tokens where they still disagree.

• Routing replay. Record the experts the engine chose for every token and force the trainer to use them. DeepSeek's Keep Routing has been in its RL pipeline since V3-0324. Rollout Routing Replay (R3), from Xiaomi and Peking University, cut the measured training–inference KL divergence from 1.5×10⁻³ to 7.5×10⁻⁴, near a dense model's 6.4×10⁻⁴. MiMo-V2-Flash, MiMo-V2.6 and MAI-Thinking-1 use it, GLM-5.3 lists "R3-style setups", and DeepSeek-V4.1-Flash concatenates the routing of each segment of a resumed sample. The GSPO authors offer their sequence-level ratio (Chapter 03) as a way to drop replay, but their replay is a different one: it copies π_old's routes within the trainer, while R3 copies the inference engine's routes into the trainer.
• Sampling-mask replay. Top-p sampling draws from the most likely tokens only, renormalized over that subset, so π_infer is computed over the subset; unless the trainer renormalizes the same way, ρ_t is wrong even with identical weights. With probabilities 0.5, 0.3, 0.15, 0.05 and p = 0.8, the engine logs 0.3 / 0.8 = 0.375 for the second token and the trainer's full softmax 0.30: a ratio of 0.8 with no error anywhere. DeepSeek's Keep Sampling Mask, MAI's top-p mask replay and MiMo-V2.6's candidate-set replay record each token's subset and apply it to the trainer's softmax.
• Deterministic kernels. DeepSeek-V4 built batch-invariant kernels (a token's output does not depend on what else shares its batch), deterministic across pre-training, post-training and inference. In GLM-5 one non-deterministic operator, a top-k selection inside its sparse attention, caused drastic RL degradation within a few steps until it was replaced by a deterministic one. GLM-5.3 reports a mean log-probability gap at the 10⁻⁷ level, and StepFun reports a difference of exactly zero for Step 5 Preview at staleness 0.
• One precision in both engines. MAI runs learner and engine in BF16 (16-bit floats; pretraining notes, Chapter 11) after finding that lower precisions gave larger gaps. DeepSeek-V4, Kimi K3 and MiMo-V2.6 roll out in 4-bit formats (FP4, and MX formats whose blocks share one power-of-two scale) and make the trainer use the same quantized weights; Chapter 09 explains how.
• Higher precision where the gap starts. MiniMax traced M1's mismatch to large activations in the output layer and computed the LM head in FP32 (June 2025). Ant's Ring-linear-2.0 aligned the engines module by module, with FP32 for the accumulated linear-attention state, the LM head, RMSNorm and the residual, and then found that "no additional algorithmic modifications are necessary".
• IcePop-style token masking. IcePop, from inclusionAI's Ling team (introduced with Ring-flash-2.0 in September 2025 and detailed in the Ring-1T report), multiplies each token's term by the mismatch ratio κ_t = π_train/π_infer, both computed with the sampling weights, and drops the token when κ_t falls outside a band. GLM-5 uses it in reasoning RL and NVIDIA's Nemotron 3 Ultra in its on-policy distillation; Ant's Ring-2.6-1T replaced the fixed band with KPop, which "replaces the uniform fixed-ratio constraint with binary KL divergence". Kimi K2.5 and K3 zero the gradient of tokens whose log-ratio to π_old leaves a band, a rule K2.5 introduced against drift from training–inference discrepancies, and MiMo-V2-Flash discards tokens that "exhibit large discrepancies".

MiMo-V2.6 also froze the router during RL after a trainable router collapsed expert load within 20 steps (at decoder layer 9: coefficient of variation 0.78 to 2.0, cold experts 0.5% to 22%). The stated reason is load balance, not engine mismatch.

Who uses it and why

The table follows the release-by-feature grid behind the cross-lab posttraining section of the atlas.

"Even small per-token logprob discrepancies compound across long rollouts and can destabilize the importance-sampling correction in off-policy RL."

"This Keep Routing operation was found crucial for RL training stability of MoE models, and has been adopted in our RL training pipeline since DeepSeek-V3-0324."

"Due to the long-tail nature of the rollout process, naive synchronous RL training introduces substantial bubbles during the rollout stage because of the severely imbalanced generation of agentic tasks, which can cause large GPU idle time."

1. Fu et al. AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning, §4.2, §5 (2025).
2. Ma et al. Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers (R3), §3-5 (2025).
3. Ling Team, inclusionAI. Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model (Ring-1T, IcePop), §2.3.2, §2.3.4 (2025).
4. Zheng et al. Group Sequence Policy Optimization, §5.3 (2025).
5. Kimi Team. Kimi k1.5, §2.6.2; Kimi K2, §3.3; Kimi K2.5, §4.4.2; Kimi K3, §4.1.2, §4.1.4.
6. DeepSeek-AI. DeepSeek-R1, §3.2.1; DeepSeek-V3.2, §3.1; DeepSeek-V4, §3.3, §5.2.1; DeepSeek-V4.1-Flash, §5.2.
7. Microsoft AI. MAI-Thinking-1 report, §3.1.3, §3.1.5, §3.6.
8. Xiaomi. MiMo-7B, §3.4; MiMo-V2-Flash, §4.4, §4.6; MiMo-V2.6, §4.1, §5.1, §5.4, §6.4.
9. Z.ai. GLM-5 report, §3.2, §4.1; GLM-5.3 blog, slime section.
10. Thinking Machines. Introducing Inkling. Meta. The Llama 4 herd.

07 Distillation and merging specialists

A lab has RL-trained separate specialists for math, code and agents: how does it turn them into one model, and why did several labs move from fine-tuning on the specialists' outputs to grading the student's own samples?

Why specialists, and why they must be merged

Reinforcement learning (RL) on one domain at a time is easier to run: each team owns its prompts and rewards, and runs proceed in parallel. Training all skills together risks what Xiaomi calls the "see-saw" effect, where improving one skill regresses another. So labs train specialists, copies of one base model each RL-trained on one domain (MAI-Thinking-1, Microsoft AI's reasoning model, used three "to enable parallel development"; DeepSeek-V3.2 used six). Shipping one model then needs a consolidation step, and there are three ways to do it:

• Off-policy distillation. The specialist writes answers and the new model is fine-tuned to copy them.
• On-policy distillation. The new model writes answers and the specialist grades every token.
• Weight merging. No training: the specialists' parameters are averaged.

In both distillations the specialist is the teacher π_T and the model being trained is the student π_S, the policy π_θ with parameters θ. They differ only in who writes the tokens.

Off-policy distillation: the teacher writes, the student copies

Knowledge distillation trains a student to imitate a teacher; Hinton et al. trained small classifiers on a large model's class probabilities, its "soft targets". For language models the simplest form samples responses from the teacher and runs supervised fine-tuning (SFT) of the student on them. Say the math teacher answers "17 × 23?" with "17 × 20 = 340, 17 × 3 = 51, so 391". SFT raises the student's probability of each of those tokens, each time given the teacher's tokens before it. The data come from another model, so this is off-policy distillation. Over prompts x and teacher responses y:

The teacher's entropy H(π_T) does not depend on θ, so SFT on teacher samples minimizes the forward KL, teacher first.

This was the 2025 default. Each DeepSeek-V3 expert "serves as a data generator for the final model"; GLM-4.5, Z.ai's mixture-of-experts flagship (pretraining notes, ch 02), distills three experts with SFT; MAI-Thinking-1 runs 4 epochs of consolidation SFT, then light RL, and StepFun's Step 3.5 Flash (February 2026) distills its domain experts into one student through their rejection-sampled outputs before its final RL. Gemma 3 distills from "a large IT teacher" without saying whether on- or off-policy.

The flaw: exposure bias

In SFT every prefix y_<t was written by the teacher; at test time the student continues its own. If it writes "17 × 20 = 34" where the teacher wrote 340, the next token follows a prefix no training example covered. This is exposure bias (ch01), and it gets worse with length: illustratively, if the student strays from what the teacher would write with probability 0.001 per token, a 10,000-token chain of thought stays on teacher-like prefixes with probability 0.999^10,000 ≈ e⁻¹⁰ ≈ 0.00005.

On-policy distillation: the student writes, the teacher grades

On-policy distillation (OPD) changes who writes the tokens. The student answers the prompt itself; the teacher reads that answer in one forward pass and, at every position, gives the probability it would have written the same token. Say the student writes "17 × 20 = 340, 17 × 3 = 41". At "340" both give 0.9 and the signal is ln(0.9/0.9) = 0; at "41" the teacher gives 0.001 against the student's 0.4, and the signal is ln(0.001/0.4) ≈ −6.0, a strong push away from the error where it happened. Figure 7.1 sets the two side by side.

Formally, the student samples y ∼ π_S(·|x) and minimizes the reverse KL, student first, at each of its own prefixes (MiniLLM, Gu et al.; GKD, Agarwal et al., samples the same way but treats the divergence as a choice: forward KL, reverse KL or a mix of the two). Here π_S(·|x, y_<t) is the student's next-token distribution, a vector in ℝ^|𝒱| over the vocabulary 𝒱:

Estimating each position's KL from the one sampled token gives the example's per-token signal, used as a fixed weight on the gradient:

sg is stop-gradient; A_t acts as a per-token advantage (ch03), so OPD runs in ordinary RL code. The per-token form ignores a token's effect on later positions; Thinking Machines calls this a "discount factor of zero".

Three properties follow:

• It trains where the student goes. Training prefixes come from the student, so it learns in the states it will visit, including those its mistakes lead to.
• It is mode-seeking. In reverse KL, ∑_v π_S(v) log(π_S(v)/π_T(v)), mass on a token the teacher almost never says costs a lot, while a teacher mode the student skips (π_S(v) = 0) costs nothing, so a small student commits to one mode. Forward KL weights terms by π_T, so missing a mode is expensive and the student spreads over all modes, including the low-probability region between them (B in the worked example below).
• It is dense. Every token gets a grade, not just the final answer. From an SFT checkpoint at 60% on AIME'24, Thinking Machines reached 70% in about 150 OPD steps, an estimated 9–30× cheaper than more SFT.

def opd_step(student, teachers, prompts, eta, R_max=None):
    """One on-policy distillation step with a per-token reverse-KL reward."""
    losses = []
    for x in prompts:
        teacher = teachers[domain(x)]          # multi-teacher: route by the prompt's domain
        y, logp_old = student.sample(x)        # y ~ pi_S(.|x); logp_old[t] = log pi_S(y_t|x,y_<t), no grad
        logp_T = teacher.logprob(x, y)         # one teacher forward pass over the student's tokens
        A = logp_T - logp_old                  # A_t = minus the sampled reverse KL; a constant (sg)
        if R_max is not None:
            A = clip(A, -R_max, R_max)         # Kimi K3 clips the per-token reward at R_max
        logp = student.logprob(x, y)           # same tokens, now with gradient
        rho = exp(logp - logp_old)             # importance ratio rho_t; equals 1 when fully on-policy
        losses.append(-(rho * A).mean())       # grad = -mean_t A_t * grad log pi_S(y_t|x,y_<t)
    loss = mean(losses)
    loss.backward()
    student.update(eta)

# Full-vocabulary variant (GKD's per-position form; DeepSeek-V4 with reverse KL): exact KL per position
def full_vocab_kl(logits_S, logits_T):         # both [|y|, |𝒱|]
    logq, logp = log_softmax(logits_S), log_softmax(logits_T)
    return (exp(logq) * (logq - logp)).sum(-1).mean()

Many teachers in one run

Multi-teacher OPD (MOPD, Xiaomi's name) routes each prompt to its domain's teacher. MiMo-V2-Flash, Xiaomi's 309B-total MoE, introduced it; a teacher may be an RL model, an SFT model "or even the student model itself", and its advantage is added to an outcome-reward advantage. GLM-5's final cross-stage OPD uses its own earlier-stage checkpoints as teachers to recover skills lost in sequential RL, and GLM-5.2 ran parallel OPD over more than ten experts. Kimi K3, Moonshot's 2.8T-total MoE, picks the teacher by domain and requested effort and clips the per-token reward.

DeepSeek-V4 "entirely replaced" its mixed RL stage with OPD, minimizing ∑_i w_i KL(π_S ‖ π_T,i) over teachers i with teacher weights w_i, each KL computed exactly over the full vocabulary ("full-vocabulary logit distillation"). Instead of storing teachers' logits it caches their last-layer hidden states and recomputes logits during training: a hidden state is d numbers per token (7,168 for V4-Pro), a logit vector |𝒱| numbers (129,280), 18 times more. V4.1-Flash's teachers may come from different development stages and architectures. Teacher counts from the coded grid (the release-by-feature table behind the cross-lab posttraining page, the atlas's posttraining section):

Weight merging: average the parameters

Weight merging averages the parameters θ_i of models fine-tuned from one base, θ = ∑_i c_i θ_i with mixing weights c_i summing to 1. Take one weight that is 0.50 in the base, 0.62 in a math specialist and 0.44 in a chat specialist: an even merge gives 0.53, so each specialist's change (+0.12 and −0.06) survives at half strength, and changes in opposite directions partly cancel. Kimi k1.5 merged a long-CoT and a short-CoT model "by simply averaging their weights".

For consolidation it fell out of favor: Xiaomi says merging and sequential training "often trade off capabilities", and DeepSeek-V4 consolidates without it, saying OPD sidesteps "the performance degradation often encountered in traditional weight-merging". Merging survives for other jobs:

• Restarting RL. DeepSeek-V4.1-Flash combines checkpoints of RL runs on different agent scaffolds to start the next RL run.
• Balancing skills. NVIDIA's Nemotron Nano 2 interpolated a reasoning-strong and a chat-strong RL checkpoint, and Bolt-74B (October 2026) SLERP-merged checkpoints twice, the second time its RLHF and SWE-RL branches (SLERP interpolates along an arc rather than a straight line).
• Method not stated. Mistral calls Medium 3.5, which puts instruct, reasoning and coding in one set of weights, its "first flagship merged model" without saying how it was merged.

Distilling into smaller models

The same two distillations make small models from large ones:

• Off-policy only. DeepSeek-R1 fine-tuned Qwen and Llama bases on 800,000 R1-generated samples, SFT only. On the same 32B base, distillation scored 72.6 on AIME 2024 (pass@1) against 47.0 after over 10K RL steps. MiMo-V2.6-Distill-Qwen-9B is Qwen3.5-9B fine-tuned on 77.4B tokens of MiMo output, lifting SWE-bench Pro from 32.0 to 44.6.
• With an on-policy phase. Qwen3 (April 2025) distilled its smaller models in two phases, off-policy on the flagships' outputs and then on-policy, matching the logits of Qwen3-32B or Qwen3-235B-A22B; on Qwen3-8B, starting from the same off-policy checkpoint, the on-policy phase used 1,800 GPU-hours against 17,920 for RL and scored higher. Inkling-Small, Thinking Machines' 276B-total, 12B-active MoE, was post-trained in part with OPD from Inkling, then got two weeks of agentic coding RL; no loss or data details are given.

Who uses it and why

"By learning from its own distribution, the student avoids the exposure bias and distribution mismatch common in off-policy methods trained on static datasets."

"Although this approach is resource-efficient, it leads to high variance in gradient estimation and often causes training instability. Therefore, we adopt full-vocabulary logit distillation in our OPD."

1. Hinton, Vinyals, Dean. Distilling the Knowledge in a Neural Network (2015)
2. Agarwal et al. On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (GKD, ICLR 2024)
3. Gu et al. MiniLLM: On-Policy Distillation of Large Language Models (2023)
4. Lu and Thinking Machines Lab. On-Policy Distillation (Connectionism, Oct 2025)
5. DeepSeek-AI. DeepSeek-R1, App. F: Distillation (2025)
6. Kimi Team. Kimi k1.5: Scaling Reinforcement Learning with LLMs (2025)
7. DeepSeek-AI. DeepSeek-V3 Technical Report (2024)
8. DeepSeek-AI. DeepSeek-V3.2 (2025)
9. DeepSeek-AI. DeepSeek-V4 Technical Report (2026)
10. DeepSeek-AI. DeepSeek-V4.1-Flash Technical Report (2026)
11. Z.ai. GLM-4.5 Technical Report (2025)
12. Z.ai. GLM-5 Technical Report (2026)
13. Z.ai. GLM-5.2 blog (2026)
14. Moonshot AI. Kimi K3 Technical Report (2026)
15. Xiaomi. MiMo-V2-Flash Technical Report (2025)
16. Xiaomi. MiMo-V2.6 Technical Report (2026)
17. Microsoft AI. MAI-Thinking-1 Technical Report (2026)
18. Gemma Team. Gemma 3 Technical Report (2025)
19. Thinking Machines. Inkling-Small (2026)
20. Meta. Muse Glimmer 30B: An On-Device Agentic Model (2026)

08 Thinking and effort control

How does one model learn to think for a few hundred tokens on an easy request and tens of thousands on a hard one, and how do labs let users turn that dial?

Reasoning models think before they answer

A chain of thought is "a series of intermediate reasoning steps" written before the answer. A reasoning model is trained, usually by reinforcement learning (RL), to write a long one between <think> tags and is rewarded on the answer that follows. DeepSeek-R1-Zero showed that the length need not be taught: RL from DeepSeek-V3-Base with only a think-then-answer template and rule-based rewards for correctness and format made its responses grow steadily, while AIME 2024 pass@1 (the share of problems solved by a single sample) rose from 15.6% to 77.9%. In DeepSeek's words the model "naturally learns to solve reasoning tasks with more thinking time".

Thinking costs output tokens and latency: DeepSeek-R1 averaged 8,793 thinking tokens per 2024 competition-math problem, under 7,000 on easy ones and over 18,000 on the hardest. Labs expose three separate controls over it:

• The switch: whether the model thinks at all.
• Effort or budget: how long it thinks when it does.
• Thinking history: whether the thinking of earlier turns stays in context for later ones.

The switch: one model, two modes

The first reasoning models shipped as separate checkpoints beside a chat model: DeepSeek-R1 beside V3, GLM-Z1-0414 beside GLM-4-0414, Kimi K2-Thinking beside K2-Instruct. From March 2025 labs began to merge them into one hybrid model with a switch in the chat template, the code that turns a conversation into tokens. The switch is a marker. The model learns it from training examples in both formats, each preceded by its marker; GLM-4.5, for instance, balanced its supervised fine-tuning (SFT) data "containing full reasoning with data lacking explicit thought processes". Schematically, with the marker at the start of the answer as in DeepSeek-V3.1:

User:       17 × 23?
thinking:   Assistant: <think> 17×20=340, 17×3=51 </think> 391
direct:     Assistant: </think> 391

Labs place the marker in different spots:

• In the system prompt. NVIDIA's Llama-Nemotron toggles reasoning with a system prompt, "detailed thinking on/off", and Gemma 4 uses a <|think|> system-prompt token.
• As a flag in the prompt. Qwen3 (April 2025) uses /think and /no_think flags, which Qwen says "eliminates the need to switch between different models".
• At the start of the answer. DeepSeek-V3.1 (August 2025) "supports both thinking mode and non-thinking mode by changing the chat template": the assistant turn starts with <think> to think or with a closed </think> to answer directly.
• As a template argument. Kimi K2.5 has a thinking=False template flag.

The merge did not always last: Qwen split its line again in July 2025, when Qwen3-2507 came as a non-thinking Instruct model and a separate Thinking model, and returned to one model that thinks by default with Qwen3.5 in February 2026.

Effort levels: a length price set by the request

A switch cannot say how much to think. Effort levels write the requested level into the prompt and make the price of length, the length penalty of chapter 04, depend on it, so the model learns a length per level.

• Levels in the prompt, training not described. gpt-oss, OpenAI's open-weight model, supports three levels set by a system-prompt line such as "Reasoning: low"; OpenAI does not say how they were trained.
• One specialist per level. DeepSeek-V4 trained distinct specialists for Non-think, Think High and Think Max with "distinct length penalties and context windows". Kimi K3 trains each domain expert at low, high and max with a per-problem token budget: a trajectory longer than c · b₀(x) tokens, where b₀(x) is estimated from the cold-start model (the SFT model that RL starts from), gets reward −1. With b₀(x) = 4,000 and c = 2, say, anything past 8,000 tokens scores −1. Kimi trains max first with a large multiplier c (Kimi writes τ, which these notes reserve for sampling temperature), then anneals c for high and low.
• One model, level in the prompt. Inkling, Thinking Machines' mixture-of-experts (pretraining notes, ch 02) 975B-total model, uses both ingredients, "changing the system message and adjusting the per-token cost", and exposes seven presets from none to max. Meta's Muse Glimmer trains strength-specific length penalties in subgroups and routes some prompts only to higher strengths, whose samples "primarily drive capability acquisition".

A numeric effort level: DeepSeek-V4.1-Flash

DeepSeek-V4.1-Flash made effort a number: the system prompt carries "Reasoning Effort: {effort}" with an integer from 1 to 100. The idea is a price per reasoning token that falls as the requested effort rises, so long answers pay off only at high effort. Each prompt gets several samples at each of a few training levels, and rewards are mean-centered within each (prompt, effort) subgroup, so "responses from different effort levels are not directly compared" (group advantages: ch03). Effort enters the reward only through the length penalty:

ℓ counts the reasoning tokens of response y, L_norm is a reference length, C_max caps the deduction, k₀ applies at the lowest training level e_min, and w is the decay width (DeepSeek writes b for e and τ for w). Figure 8.1 draws the penalty at two levels with illustrative constants. The API maps low, high and max to e = 50, 75 and 100.

def effort_rl_step(policy, prompts, efforts=(25, 50, 75, 100), m=4,
                   k0=0.4, w=25.0, L_norm=10_000, C_max=0.5):
    """Effort-conditioned rewards in the style of DeepSeek-V4.1 (illustrative constants)."""
    batch = []
    for x in prompts:
        for e in efforts:                                  # one subgroup per (prompt, effort)
            k = k0 * exp(-(e - min(efforts)) / w)          # penalty coefficient k(e)
            sub = []
            for _ in range(m):
                y = policy.sample(system=f"Reasoning Effort: {e}", prompt=x)
                ell = count_reasoning_tokens(y)
                r = task_reward(x, y) - min(C_max, k * ell / L_norm)
                sub.append((y, r))
            baseline = sum(r for _, r in sub) / m          # mean of this subgroup only
            batch += [(x, e, y, r - baseline) for y, r in sub]   # advantage A
    policy.update(batch)                                   # clipped policy-gradient step (ch03)

Hard budgets

A hard budget needs no effort-conditioned reward: thinking is cut off at a token count. Qwen3 (April 2025) halts thinking at a user-set length by inserting a stop-thinking instruction, an ability Qwen says was not explicitly trained, and NVIDIA's Nemotron Nano 2 (August 2025) put about 5% deliberately truncated reasoning traces into SFT so the model can answer after its thinking is cut off at the budget.

Always-on thinking: effort replaces the switch

Some 2026 models dropped the non-thinking mode. Kimi K3 "always has thinking enabled" with low, high and max effort, and GLM-5.3 says "Disabling thinking is no longer supported", telling users of the old off switch to request low effort instead. Neither lab states why. Qwen3.8 (August 2026) followed, offering xhigh, medium and low effort with no way to turn thinking off in its open 2.4T checkpoint. The cheapest setting is now low effort. Figure 8.2 places each style on a timeline.

Keeping thinking across turns

The third control is about memory rather than length. Before each response the chat template rebuilds the history and decides whether earlier thinking goes back in. If an agent thinks, calls a tool and thinks again, dropping the first block leaves the second with the call and its result but not the reasoning behind the call. For an agent that makes dozens of tool calls, dropping thinking at every tool result forces it to rebuild its plan each time; keeping it costs context, about 30,000 extra tokens for 30 calls with 1,000 thinking tokens each (illustrative). Releases follow three policies:

• Drop it. Chat templates used to delete earlier reasoning: R1 dropped it once the next round of messages arrived, and gpt-oss's guide says "the reasoning traces from past assistant turns should be removed".
• Keep it through tool calls, drop it at a new user message. DeepSeek-V3.2 found R1's rule "results in significant token inefficiency" and kept reasoning across tool outputs until a new user message.
• Keep it in the history. DeepSeek-V4 keeps it across user turns too in tool-calling contexts. MiniMax's M2 (October 2025) is an "interleaved thinking model" whose earlier thinking must stay in the history. GLM-4.7 added Preserved Thinking for coding agents, Kimi K2.6 an opt-in preserve_thinking flag, Qwen3.6 (April 2026) was "additionally trained to preserve and leverage thinking traces from historical messages", and Kimi K3 was "trained in the preserved thinking history mode".

The coded grid (the release-by-feature table behind the cross-lab posttraining page, the atlas's posttraining section) records each release:

Who uses it and why

"Alongside advances in model architecture and hardware, the number of output tokens is another key determinant of serving cost and, consequently, of the cost–quality trade-off in real-world applications."

"Preserved Thinking: In coding agent scenarios, GLM-4.7 automatically retains all thinking blocks across multi-turn conversations, reusing the existing reasoning instead of re-deriving from scratch."

1. Wei et al. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models (2022)
2. DeepSeek-AI. DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (2025)
3. Kimi Team. Kimi k1.5: Scaling Reinforcement Learning with LLMs (2025)
4. OpenAI. gpt-oss-120b & gpt-oss-20b Model Card (2025)
5. DeepSeek-AI. DeepSeek-V3.1 model card (2025)
6. DeepSeek-AI. DeepSeek-V3.2 (2025)
7. DeepSeek-AI. DeepSeek-V4 Technical Report (2026)
8. DeepSeek-AI. DeepSeek-V4.1-Flash Technical Report (2026)
9. Z.ai. GLM-4.5 Technical Report (2025)
10. Z.ai. GLM-4.7 blog (2025)
11. Z.ai. GLM-5.3 blog (2026)
12. Moonshot AI. Kimi K2.6 model card (2026)
13. Moonshot AI. Kimi K3 Technical Report (2026) and model card
14. Google. Gemma 4 model card (2026)
15. Xiaomi. MiMo-V2.6 Technical Report (2026)
16. Thinking Machines. Introducing Inkling (2026)
17. Meta. Muse Glimmer 30B: An On-Device Agentic Model (2026)

09 Quantization-aware posttraining

How do labs ship models whose expert weights are stored in 4 bits while keeping the accuracy those models had in 16 bits?

Why serve in 4 bits

When a model generates text, each new token reads every active weight from GPU memory (pretraining chapter 00). The bits stored per weight therefore set both how large a model fits on a GPU and how many bytes each decode step moves. The weights alone take

A toy model with 10B parameters takes 10 × 10⁹ × 16 / 8 = 20 GB in BF16 and 5 GB at b = 4, a quarter of the memory and of the weight bytes each decode step reads.

OpenAI's gpt-oss-120b, a mixture-of-experts (MoE) model with 116.8B parameters (5.1B active per token), keeps 114.71B of them in MoE weights stored at 4.25 bits, 4 bits per weight plus a shared scale (explained below): 114.71 × 10⁹ × 4.25 / 8 = 60.9 GB. The other 2.12B parameters in BF16 add 4.2 GB, for 65.2 GB (60.7 GiB), close to the 60.8 GiB checkpoint OpenAI lists, which fits on a single 80GB GPU. In BF16 the model would take 116.83 × 10⁹ × 2 bytes = 234 GB.

The 4 bits go where the bytes are: MoE weights are 90+% of gpt-oss's parameters, and Kimi says expert weights "dominate the model's parameter memory" in K3. Attention, routers and embeddings stay in higher precision (gpt-oss config); K3 also keeps shared experts and latent projections there.

What a 4-bit weight stores

Four bits give only 16 codes, too few to serve both a block of weights near 0.01 and one near 1. So every 4-bit format stores a 4-bit code per weight plus a scale shared by a small block, and a stored weight is code × scale. This is block scaling: divide the block by its scale so that its largest value lands near the top of the code range, round, and multiply the scale back. Formats differ in which codes they allow and how they store the scale.

Integer codes are evenly spaced. Floating-point codes have a sign bit, exponent bits that set the range and mantissa bits that set the precision, so they crowd near zero and thin out away from it (details in pretraining chapter 11); BF16 is the usual 16-bit training format and FP32 the 32-bit one. Four formats appear in this chapter:

• FP8 E4M3, the 8-bit reference: 4 exponent and 3 mantissa bits, so rounding moves a value by at most about 6%.
• INT4: evenly spaced integer codes. In LLM-QAT's symmetric min-max scheme, for example, a group's scale is α = max|w| / 7 and each weight becomes an integer from −7 to 7, the largest exactly 7.
• FP4 (E2M1): a sign bit, 2 exponent bits and 1 mantissa bit, giving magnitudes 0, 0.5, 1, 1.5, 2, 3, 4 and 6. Unlike INT4 the grid is uneven: the step is 0.5 up to 2, then 1 up to 4, then 2, so above 1 one rounding moves a value by up to 20% (5 lies 1 from both 4 and 6), and by more below 1. The next two formats store E2M1 codes.
• MXFP4: the microscaling (MX) standard of the Open Compute Project (OCP) groups 32 E2M1 elements with one shared 8-bit E8M0 scale, an exponent-only number, so the scale is a power of two and storage is (32 × 4 + 8) / 32 = 4.25 bits per value.
• NVFP4: NVIDIA's format keeps E2M1 elements but uses blocks of 16 and an E4M3 block scale, which can be fractional, plus one FP32 scale per tensor to cover the range E4M3 lacks, for 4.5 bits per value.

Figure 9.1 draws the two FP4 layouts to scale, above the E2M1 grid they share.

Choosing the scale. Unlike INT4's min-max scale, an MXFP4 scale must be a power of two. For a block of weights w₁ … w₃₂, the MX paper's example conversion (Algorithm 1, which follows the OCP specification) puts the block maximum in E2M1's top power-of-two range and clamps anything beyond 6:

A power-of-two scale cannot put the block maximum exactly at 6, so values between 6 and 8 times the scale clamp. Doubling the scale would avoid that, but then no block would use the top grid values 4 and 6 and every weight would sit on a grid twice as coarse. The MX paper's rule aims at "full utilization of the element data format's exponent range" and accepts occasional clamping. NVFP4's fractional scale can put the block maximum right at 6, and its smaller blocks follow local ranges more closely, at a quarter bit more per value. The worked example below shows both effects.

Rounding after training, or training with the rounding

The baseline is post-training quantization (PTQ): train in 16 bits, then round the finished weights once, as in the worked example. Jacob et al. found that PTQ works for large models but loses accuracy on small ones, for example when outlier weights force a coarse grid on all the others, and the LLM-QAT authors found that PTQ methods for LLMs "perform well down to 8-bits" but break down below. PTQ can work at 4 bits (NVIDIA reports 1% or less accuracy loss for DeepSeek-R1-0528 in NVFP4), but Kimi says low-bit quantization often causes substantial drops for thinking models because their outputs are so long.

Quantization-aware training (QAT) puts the rounding inside training, so the network can adapt to it. The optimizer keeps FP32 master weights w, and every forward pass uses a fake-quantized copy ŵ: each block is rounded to its 4-bit grid and converted straight back to a float. The loss is then the loss of the model that will be served. For each weight of a block:

Q is a staircase. It is flat between rounding points, so dQ/dw = 0 almost everywhere and the exact gradient would never change w. The straight-through estimator (STE), named by Bengio et al. after a proposal by Hinton, back-propagates through the rounding as if it were the identity:

Bengio et al. call it "clearly a biased estimator". It works because the master weights stay in floating point "so that they can be easily nudged by small amounts" (Jacob et al.): small updates add up in FP32 until a weight crosses a rounding boundary. In the worked example's MXFP4 block every master value from 0.3125 to 0.4375 is served as 0.375, so a weight of 0.33 nudged by −0.001 per step (scale fixed) is served as 0.375 for 17 steps and as 0.25 from the 18th.

E2M1 = tensor([0.0, 0.5, 1.0, 1.5, 2.0, 3.0, 4.0, 6.0])   # FP4 magnitudes

def mxfp4_fake_quant(w):                   # w: [n_blocks, 32] FP32, blocks along the input dim
    amax = w.abs().amax(-1, keepdim=True).clamp(min=1e-30)  # [n_blocks, 1]
    s_blk = 2.0 ** (floor(log2(amax)) - 2)  # power of two (E8M0); 6 = 1.5 * 2**2
    r = (w.abs() / s_blk).clamp(max=6.0)    # values above the grid clamp to 6
    q = E2M1[(r[..., None] - E2M1).abs().argmin(-1)]  # nearest FP4 magnitude
    return sign(w) * q * s_blk              # dequantized w_hat: [n_blocks, 32]

def ste_quant(w):                           # forward: w_hat; backward: identity
    return w + (mxfp4_fake_quant(w) - w).detach()

def qat_step(model, w_master, batch, opt):  # w_master: FP32 expert weights
    w_hat = ste_quant(w_master.reshape(-1, 32)).reshape(w_master.shape)
    loss = model.loss(batch, expert_weights=w_hat)  # forward sees 4-bit values
    loss.backward()                         # dL/dw_master = dL/dw_hat (STE)
    opt.step(); opt.zero_grad()             # FP32 update; small steps accumulate

INT4 weight-only QAT: the Kimi K2 line

Kimi K2-Thinking, Moonshot's MoE with 1T total and 32B active parameters, applies QAT during post-training with INT4 weight-only quantization of the MoE components. "Weight-only" means activations and arithmetic stay in 16 bits, so the gain comes from moving fewer bytes: Kimi reports a roughly 2× generation speedup, with every reported benchmark run at INT4. The checkpoint config stores symmetric 4-bit integers in groups of 32 weights and leaves attention, shared experts, the dense MLP and the output head unquantized. K2.5, K2.6 and K2.7-Code reuse the method.

Other labs place INT4 QAT elsewhere. Z.ai's GLM-5 applies it in the SFT stage, using one kernel for training and for the final weight conversion so the two match bit for bit. Google's Gemma 3 uses a short QAT run after training: about 5,000 QAT steps per model, with the unquantized model's probabilities as targets.

MXFP4 in post-training

gpt-oss was post-trained with its MoE weights quantized to MXFP4, ships them in MXFP4, and ran all its evaluations in MXFP4.

DeepSeek-V4 (V4-Pro: 1.6T total, 49B active) added MXFP4 QAT in post-training in two places: the MoE expert weights, and the query-key path of a small scoring module inside its sparse attention (the indexer, which picks the cached tokens each query reads). Teacher and reference models get QAT too.

For the expert weights, DeepSeek turns the FP4 values back into FP8 for computation and calls this lossless. An FP4 value times its power-of-two scale has one mantissa bit, which E4M3 (three mantissa bits, two more exponent bits) stores exactly: FP4 value 1.5 with scale 2⁻³ is 0.1875 = 1.5 × 2⁻³, an exact E4M3 value. The one condition is range: each FP8 block of 128 × 128 weights has a single scale, so the FP4 scales of the 32-value blocks inside it must not spread too far apart, which DeepSeek checked empirically for its weights. QAT therefore runs in DeepSeek's existing FP8 framework, with gradients of the FP8 weights passed straight to the FP32 masters (the STE).

Moonshot's Kimi K3 (2.8T total, 104B active) moved from the K2 line's INT4 to MXFP4 expert weights with MXFP8 activations (the 8-bit MX format) "for broad hardware compatibility", and runs QAT through all of post-training, SFT and RL. MiMo-V2.6, Xiaomi's MoE family (the Pro model has 1.02T total, 42B active), starts MXFP4 QAT earlier, in mid-training, and serves its experts in MXFP4 during RL rollouts.

NVFP4 and the FP4 KV cache

Two of the labs added to the atlas later ship NVFP4 weights. MiniMax's M3.1-Flash quantizes its routed experts and attention paths during training and stores the experts in NVFP4. NVIDIA ships Nemotron 3 Super and Ultra in NVFP4 after post-training quantization alone (Ultra keeps attention in BF16); for Nemotron Nano 2 VL (October 2025) and the Nemotron 3.5 models it adds a short repair step it calls quantization-aware distillation (QAD), training the PTQ model to match the BF16 model's outputs, much as Gemma 3's QAT used the unquantized model's probabilities as targets. Bolt-74B extends QAD to 2-bit experts (2.0625 bits per weight).

DeepSeek-V4.1-Flash, a multimodal MoE with 552B backbone parameters, applies the NVFP4 layout to its main KV cache (the keys and values kept for past tokens, pretraining chapter 03). QAT in post-training lets it store the cache as E2M1 with one E4M3 scale per 16 channels and no global scale, which it can drop because cache values stay far inside the range: at most about 22.6, against the format's 448 × 6 = 2688. The main cache ends up nearly half the size of V4's FP8 cache.

Why the rollouts must be quantized too

In RL, two engines evaluate the same weights. The inference engine samples a response and logs π_infer(y_t | x, y_<t), the probability it gave each sampled token; the trainer recomputes the same probability, π_train. If the engine serves FP4 experts while the trainer evaluates BF16 weights, the mismatch ratio κ_t = π_train / π_infer differs from 1 before any update, and the clipped objective treats numerical noise as policy change. Illustratively, trainer and sampler probabilities of 0.30 and 0.25 give κ = 1.2, already at the edge of a clip range of ε = 0.2. This is the training-inference mismatch of chapter 06; its quantization side is covered here. QAT removes most of it, since the trainer's forward pass already uses the quantized values, and the labs give the sampler the same quantized weights:

• DeepSeek-V4 samples rollouts with native FP4 weights so that sampling is "fully consistent with online deployment".
• Kimi K3 states that rollout and training share one quantization scheme.
• MiMo-V2.6 quantizes and dequantizes its experts after every update, following the numeric rules of its MXFP4 rollout kernels, so both engines hold identical expert weights.

Quantized rollouts also mean RL optimizes the model that users will run.

Who uses it and why

"However, thinking models use excessive decoding lengths, and thus quantization often results in substantial performance drops."

"To achieve inference acceleration and reducing memory traffic at deployment, we introduce Quantization-Aware Training (QAT) (Jacob et al., 2018) during the post-training stage"

"During RL, rollout and training share the same quantization scheme — eliminating the train–inference mismatch."

1. Jacob et al., Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference (2017), arXiv:1712.05877
2. Bengio, Léonard and Courville, Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation (2013), arXiv:1308.3432
3. Liu et al., LLM-QAT: Data-Free Quantization Aware Training for Large Language Models (2023), arXiv:2305.17888
4. Rouhani et al., Microscaling Data Formats for Deep Learning (2023), arXiv:2310.10537
5. Alvarez et al. (NVIDIA), Introducing NVFP4 for Efficient and Accurate Low-Precision Inference (2025)
6. OpenAI, gpt-oss-120b and gpt-oss-20b model card, §2.1 and Table 1; Hugging Face card and config.json
7. Moonshot AI, Kimi K2-Thinking model card, §4, and config.json; Kimi K2.7-Code card, §4
8. Moonshot AI, Kimi K3 technical report, §4.1.1 and §4.1.4; Kimi K3 blog
9. DeepSeek-AI, DeepSeek-V4 technical report, §5.2.1; DeepSeek-V4-Pro card
10. DeepSeek-AI, DeepSeek-V4.1-Flash technical report, §2.4.4
11. Xiaomi, MiMo-V2.6 technical report, §3.2, p.21 and §6.4
12. Z.ai, GLM-5 technical report, §2.4.3
13. Gemma Team, Gemma 3 technical report, §2.3

10 Safety, calibration and release practice

How do labs train a model to refuse what it should, answer what it can and admit what it does not know, and how do they decide that its weights are safe to publish?

The chapter takes the question in three parts: training a model to refuse the right requests, deciding whether its weights are safe to publish, and training it to say "I don't know".

Two ways to fail

Safety training steers between two errors:

• Under-refusal, which Microsoft AI calls unsafe compliance: fulfilling a request that the policy forbids.
• Over-refusal: declining a legitimate request.

Benchmarks measure both sides. FORTRESS, in Thinking Machines' table for its Inkling model, gives each model two scores: the share of harmful requests it refuses and the share of benign look-alikes it still answers.

• DeepSeek-V4-Pro refuses 36.0% of the harmful set and answers 98.5% of the benign set.
• Claude Fable 5 refuses 96.0% and answers 55.1%.
• Inkling sits between them, at 78.0% and 95.9%.

A model can usually raise one number by giving up some of the other, so labs train and report both.

The baseline: refusal data and a safety reward

The common recipe has two parts: SFT data that shows refusals for harmful prompts and full answers for borderline ones, and a safety reward in RL. Labs fill in the parts differently:

• Filtered data. Google filters Gemma 3's pretraining data for safety, drops post-training examples with unsafe or toxic outputs, and uses SFT and RLHF (RL from human feedback) "to steer the model away from undesirable behavior"; OpenAI filtered chemical, biological, radiological and nuclear (CBRN) data out of gpt-oss's pretraining.
• A safety reward model. DeepSeek's reasoning model DeepSeek-R1 trains a point-wise safety reward model, which scores each response on its own instead of comparing pairs, on 106,000 prompts whose responses are labeled safe or unsafe.
• Generated attacks. Moonshot's Kimi K2 grows its safety RL prompts automatically: an attack model turns human-written seed prompts into jailbreaks (prompts crafted to get around refusals), the target model answers, and a judge model marks each exchange as a success or failure against a rubric.

Microsoft AI's reasoning model MAI-Thinking-1 builds both failure modes into its rewards. Each safety prompt is labeled harmful (target: refuse or partially refuse) or borderline (target: a bounded, helpful answer "rather than hedging or refusing"). Judges score policy compliance, whether the response matches its target, and style. A weighted average of such scores can stay positive for a response that breaks policy: with illustrative equal weights on compliance and style, a fluent answer that breaks policy still earns 0.5 × 0 + 0.5 × 1 = 0.5. So MAI gates the reward: a non-compliant rollout gets the minimum reward whatever its other scores.

Deliberative alignment in gpt-oss

OpenAI argues that models trained on labeled examples must infer the safety rules indirectly, which generalizes poorly. Deliberative alignment teaches the written rules themselves, in two stages:

1. SFT: learn to recall the spec. A reasoning model trained only for helpfulness gets the relevant part of the safety specification in its system prompt and writes (prompt, chain of thought, answer) examples whose reasoning cites the spec. A judge model filters them, the spec is deleted from the prompts, and SFT on the result teaches the model to recall the spec unprompted (context distillation).
2. RL: reward safe answers. The same judge, which reads the spec, rewards safe answers. The chain of thought is hidden from the judge to reduce the chance of encouraging deceptive reasoning: if the reward depended on the reasoning text, RL could teach the model to write reasoning that looks compliant without reflecting how it actually decides.

OpenAI reports that its o1 models, trained this way, advanced the Pareto frontier of refusing jailbreaks (StrongREJECT) against not over-refusing benign prompts (XSTest), compared with GPT-4o and other models: o1 led on StrongREJECT while staying close to the best XSTest score. gpt-oss uses deliberative alignment to refuse disallowed content, resist jailbreaks and follow the instruction hierarchy (system messages outrank developer messages, which outrank user messages), and OpenAI put no direct optimization pressure on its chain of thought.

Refusals do not survive fine-tuning

Open weights change the question. Anyone can fine-tune a released model and train its refusals away, so labs test what the model can do with safeguards removed. OpenAI built adversarially fine-tuned versions of gpt-oss-120b: helpful-only (RL-trained to comply with unsafe requests instead of refusing them), with extra biology and cyber training. Its Safety Advisory Group concluded that even these did not reach High capability in either area under its Preparedness Framework, OpenAI's process for tracking capabilities that could cause severe harm. OpenAI also ran its biology evaluations on other open models and found another open model scoring at or near gpt-oss-120b on most of them. Thinking Machines (TML) tested Inkling with internal evaluations, four external testing organizations and helpful-only fine-tunes, and concluded that releasing it was unlikely to add material risk beyond existing open-weight models. Besides OpenAI's fixed thresholds, both decisions ask a relative question: does this release move the open-weight frontier in a dangerous domain?

Holding the weights back

Cyber capability drives the newest decisions. Z.ai added vulnerability-discovery data and environments to GLM-5.3's post-training and saw cyber capability grow "faster than we expected": ExploitBench rose from 24.4% to 54.4% over GLM-5.2. It launched the model to API and subscription users and held the weights for two weeks of safety evaluation and hardening. Moonshot launched Kimi K3 on 16 July 2026 with weights promised by 27 July; its report adds a cyber evaluation and a joint assessment by the UK AI Security Institute and NIST's CAISI, without saying whether they came before the release decision.

For Kimi K3.1 Pro, a further post-trained K3, Moonshot traced rising vulnerability-research scores to cyber content in the mid-training corpus and cyber tasks in the RL mix, and removed both for the open-weight release. A checkpoint trained without the removal solves 90.5% of Moonshot's in-house xdev-bench-hard exploit tasks at pass@3 (solved in at least one of three attempts); the released model solves 81.0%, which Moonshot says remains behind frontier proprietary models. Moonshot promises the weights by 4 November.

TML describes staging as a general tool: inference access for a limited group, monitored general access and hosted fine-tuning can each come before open weights, and each stage should widen access "only when the evidence supports it". Inkling-Small followed a short version: TML showed preview results with Inkling on 15 July and released the weights on 30 July, after testing.

Figure 10.1 lays these checks out in the order a release meets them, from data filtering to the license, with the loop through which evidence of risk sent a model back into training or delayed its weights.

Licenses

The license is the last release choice: it sets what others may do with the weights. Labs have moved in both directions:

• Toward or staying with Apache 2.0. Gemma 4 moved from the custom Gemma Terms of Use to Apache 2.0, which Google presents as a response to developer feedback. gpt-oss and Inkling use Apache 2.0, gpt-oss with an added usage policy. Mistral went the same way in January 2025, renewing its commitment to Apache 2.0 for general-purpose models.
• Toward custom licenses. GLM-5.3 replaced the MIT license of earlier GLM releases with a custom glm-5.3 license (GLM-5.3-Flash stayed MIT), and Kimi K3 replaced the K2 line's Modified MIT with a custom Kimi K3 License; neither lab gives a reason in its release posts. Qwen kept Apache 2.0 for most models but gave the 2.4T Qwen3.8 its own license, which requires a separate license for large model-as-a-service businesses, and StepFun moved from Apache 2.0 to a StepFun Model License with Step 5 Preview.

Among the labs added later, MiniMax moved furthest, one release at a time: Apache 2.0 for M1, a modified MIT for M2, a non-commercial license for M2.7 and a community license for M3.

Calibration: rewarding "I don't know"

Hallucination is partly an incentive problem. Kalai et al. show that under binary grading (1 for a right answer, 0 for a wrong one or for "I don't know") abstaining is never optimal, so a model trained or selected on such scores learns to guess. On SimpleQA, a set of short fact-seeking questions, gpt-oss-120b answers 16.8% correctly and 78.2% incorrectly.

The fix is to make wrong answers cost something. Score a correct answer 1, an abstention 0 and a wrong answer −φ, where φ ≥ 0 is the penalty. A model whose belief that its answer is right is p expects, if it answers,

Abstaining earns 0, so answering beats abstaining when p − φ(1 − p) > 0, that is when p > φ / (1 + φ). With φ = 1 a model should answer only above 50% confidence, and with φ = 3 only above 75% (the worked example below tabulates more). Labs put the penalty into rewards or into data.

Inkling's largest targeted dataset for this is short-form factual QA with abstention-aware rewards, built so that the best policy answers when confident and otherwise says "I don't know" or hedges. Some prompts encourage hedging and others forbid it, so the model also learns to follow the user's preference. TML does not publish the reward values. Gemma takes a data route: its Gemma 3 and Gemma 4 reports say that adding examples that encourage hedging and refusals "to minimize hallucinations" improved factuality scores.

Microsoft AI's MAI-Thinking-1 grades factual answers on two axes, correctness and confidence, into five classes: confident-correct gets the highest reward, a confident hallucination the steepest penalty and an abstention a neutral reward, while unconfident-but-correct answers get a reduced reward "to discourage over-hedging". MAI does not publish the weights. The box below sketches the shape of such rewards, with illustrative values.

# illustrative reward values; labels come from a judge
def factual_reward(label, phi):
    return {"correct": 1.0, "partial": 0.0,
            "abstain": 0.0, "incorrect": -phi}[label]

def should_answer(p, phi):        # p: belief answer is right
    return p - phi * (1 - p) > 0  # i.e. p > phi / (1 + phi)

def gated_safety_reward(scores, mix, r_min):  # MAI-style gate
    if not scores["policy_compliant"]:
        return r_min              # violation: minimum reward
    return sum(c * scores[k] for k, c in mix.items())

Forecasts, where the model states a probability q for an event, are graded with proper scoring rules such as the Brier loss (q − o)² for outcome o ∈ {0, 1}. A forecaster who believes 0.7 and reports 0.7 expects a loss of 0.7 × 0.3² + 0.3 × 0.7² = 0.21; reporting 1.0 instead expects 0.3 × 1² = 0.30, so overstating confidence costs; reporting q = p is optimal (chapter 04). TML trained Inkling and Inkling-Small with RL against proper scoring rules on resolved real-world forecasting questions, without naming the rule.

Evaluations now use the same accounting. AA-Omniscience, Artificial Analysis's benchmark of 6,000 knowledge questions, reports an Omniscience Index OI = 100 · (c − i) / N, with c correct and i incorrect answers out of N questions. Partial answers and abstentions count as neither correct nor incorrect, so they score 0, and the penalty is φ = 1.

Who uses it and why

"Because open weights can be customized to strip a model of its safety training, refusal behavior cannot be treated as a durable safeguard for an open-weight release."

"First, we aim not to release open weights that would extend the absolute frontier of cyber capability."

"The largest is short-form factual QA with abstention-aware rewards: answering only pays off when the model is likely to be right, so the optimal policy is to answer when confident and otherwise say “I don’t know” or give a hedged best guess."

1. Guan et al., Deliberative Alignment: Reasoning Enables Safer Language Models (2024), arXiv:2412.16339
2. Kalai et al., Why Language Models Hallucinate (2025), arXiv:2509.04664
3. Jackson et al., AA-Omniscience: Evaluating Cross-Domain Knowledge Reliability in Large Language Models (2025), arXiv:2511.13029; Artificial Analysis model pages for GLM-4.7 and GLM-5
4. OpenAI, gpt-oss-120b and gpt-oss-20b model card, §3, §4.4, §4.5 and §5.1
5. Thinking Machines, A Safe Path to Open Weights; Introducing Inkling; Inkling-Small; Inkling model card, §6
6. Moonshot AI, Introducing Kimi K3.1 Pro; Kimi K3 blog; Kimi K3 technical report, §6.2.2; Kimi K3 card, §7; Kimi K2 technical report, §3.2.1
7. Z.ai, GLM-5.3 blog and model card; GLM-5.3-Flash card; GLM-5 technical report, §1
8. Google, Gemma 3 technical report, §7.2; Gemma 4 technical report, §3 and §5; Gemma 4 launch blog
9. DeepSeek-AI, DeepSeek-R1 report, safety reward model (p.7)
10. Microsoft AI, MAI-Thinking-1 technical report, §3.4.3 and §3.4.4

11 RL scale and cost

How much compute do labs now spend on RL after pretraining, in what units do they report it, and why does generating rollouts take most of the hardware?

Five units for one quantity

An RL run repeats one loop: sample a batch of rollouts, score them, update the weights. Labs report how much of that loop they ran in five different units, and each hides something:

• Steps count optimizer updates, but their size varies: a step of DeepSeek-R1-Zero held 512 samples and a step of MiMo-V2.6 holds 25,088.
• Rollouts count sampled responses or agent trajectories, but their lengths vary: MiMo-V2.6's data sources differ 90-fold in mean generated tokens.
• GPU-hours count hardware time, but depend on the chip.
• Dollars fold everything into a price the reader cannot audit, unless the hardware comes with it. MiniMax-M1's full RL run took three weeks on 512 H800 GPUs and cost $534,700 in rental, which is 512 × 21 × 24 ≈ 258K GPU-hours at about $2.07 each, close to the $2 per GPU-hour DeepSeek-V3's report assumes.
• Share of pretraining compares two stages of one model.

Many reports give no number at all: Kimi K3.1 Pro, Moonshot's October 2026 model, has "substantially larger post-training compute" than K3, and GLM-5.3 spent "more compute" than GLM-5.2. When numbers are given, the units convert along one chain, and each link needs one extra number that a report may leave out:

1. Steps → rollouts: multiply by samples per step (prompts × group size G).
2. Rollouts → tokens: multiply by the mean tokens per rollout.
3. Tokens → GPU-hours: multiply by FLOPs per token and divide by the FLOP rate a GPU sustains, which differs widely between generation and training (below).
4. GPU-hours → dollars: multiply by the price per GPU-hour.
5. GPU-hours → share: divide by the pretraining GPU-hours on the same chip.

From 0.19% to more than a tenth

Share of pretraining is the unit that shows the trend. DeepSeek's reports give a rare series from one lab, worked through in chapter 00 and plotted in Figure 11.1A: posttraining took 0.19% of DeepSeek-V3's pretraining GPU-hours; R1, trained from the same base, used 147K GPU-hours, 5.5% of that pretraining; and V3.2's posttraining budget was more than 10% of the pre-training cost. That is a more than fiftyfold rise in under a year (10 / 0.19 ≈ 53), though the last point is a cost share and a lower bound.

Rollouts and log-linear returns

What does more RL buy? Thinking Machines and Microsoft AI report the same shape: the score rises by a constant amount each time the number of rollouts doubles. That is a log-linear curve,

Suppose b = 0.03. A run that gains 0.03 between 1M and 2M rollouts needs 4M for the next 0.03 and 8M for the one after: doubling from M to 2M costs M new rollouts, so each gain of b costs as many rollouts as all training before it, and a gain Δ from M₀ needs M₀ · 2^Δ/b rollouts in total. The curve is a fit over the range a lab reports, not a law that says how far it continues.

Thinking Machines reports Inkling's RL in rollouts: over 30M across two long continuous runs, which used the majority of post-training compute. Its score on a held-out set of reasoning evals rose from 0.264 at the SFT start to 0.356 at the released checkpoint, about 26.5 million rollouts, and "improved log-linearly throughout"; the blog does not say what the rollouts past the released checkpoint were. MAI-Thinking-1, Microsoft AI's reasoning model, likewise reports "sustained log-linear climbs over many thousands of steps", and MiMo-V2.6 plots its climb against dollars: DeepSWE average@3 (the mean score over 3 attempts) for Pro rose from 58.4 to 72.6 over the run.

Why generation takes most of the hardware

A token costs about 2·N_a FLOPs to generate, one forward pass through the N_a active parameters (the weights one token actually uses in a mixture-of-experts model), and about 6·N_a to train on, forward plus backward (derived in the pretraining notes' chapter 00). By arithmetic alone the learner needs three times the hardware of generation. MAI reports the opposite split: its largest RL job put 4,096 of 4,864 GB300 chips on inference and 768 on the learner, 84% against 16%.

Decode explains the gap. Generation has two phases (pretraining notes chapter 00):

• Prefill processes tokens that are already known, such as the prompt, in one parallel pass.
• Decode then emits one new token per sequence per forward pass. Each pass reads the weights and the sequence's whole KV cache (the stored keys and values of all earlier tokens, which attention reads at every step) from GPU memory, so decode is limited by memory bandwidth and runs far below peak arithmetic.

Training processes all tokens of a trajectory at once, like prefill, and keeps the arithmetic units busy. If both of MAI's pools were fully busy on the same tokens, each inference chip would sustain (2 / 4,096) ÷ (6 / 768) = 1/16 of a learner chip's FLOP rate. This is illustrative only: it ignores prefill, extra log-probability passes and idle time.

def rl_chip_split(tokens_per_s, n_active, peak_flops, mfu_decode, mfu_train):
    """Chips needed to generate tokens and to train on them at the same rate.
    mfu: model FLOPs utilization, the achieved share of peak arithmetic."""
    gen_flops = 2 * n_active * tokens_per_s              # forward pass only
    train_flops = 6 * n_active * tokens_per_s            # forward + backward
    gen_chips = gen_flops / (peak_flops * mfu_decode)    # decode: memory-bound, low mfu
    train_chips = train_flops / (peak_flops * mfu_train) # large matmuls, high mfu
    return gen_chips, train_chips

# Illustrative: if decode reaches 1/16 of the learner's utilization,
# gen_chips / train_chips = (2 / 6) * 16 ≈ 5.3, the ratio of MAI's 4,096 : 768.

Agent rollouts add to the bill: 100K-token contexts make each KV cache large, tool calls stall sequences, and the slowest trajectories hold a batch open. Partial rollouts, which pause unfinished trajectories and resume them after the update (chapter 06), keep the GPUs busy at a price: MiMo-V2.6 pays for "re-prefill", rebuilding each resumed sequence's KV cache under the new weights, and uses a large batch to amortize it.

MiMo-V2.6, Xiaomi's September 2026 release, reports a split of cost rather than chips, and it is more even (Figure 11.1B): rollout 43.8%, training 43.5%, grader 12.7% for Pro. The report credits its large batch for efficient rollout: "With HBM mostly occupied by the running batch, we can keep a high arithmetic intensity to fully use the GPU in rollout." HBM is the GPU's main memory, and arithmetic intensity is FLOPs per byte read from it. A decode step over many sequences reads each weight once for all of them, so it does more arithmetic per byte; that is one reason the split can sit closer than MAI's chip split to the 1 : 3 FLOP ratio of generation to training, though the report makes no such comparison and does not split the training share further. The grader, an LLM agent that judges trajectories within each group, takes 12.7% by itself.

Figure 11.1 plots the DeepSeek share series from above (A) next to MiMo's cost split (B).

When a fix means rerunning RL

Few, huge steps make late problems expensive. After release, MiMo-V2.6 was found to repeat tool calls and to issue too many of them; its RL penalty for such "flooding" zeroed a rollout only above 32 tool calls in one turn, too loose a threshold. Lowering it to 8 worked in a small experiment but took about 20 steps to act: "Applying this fix at full scale would therefore require restarting 20 steps of MixRL, at an estimated cost of $2.31 million." (MixRL is the mixed RL run above.) Xiaomi instead trained a repetition-specialist RL teacher for 12 steps on about 7,000 examples, rolled its final multi-teacher on-policy distillation run (MOPD: the model learns from several teachers' token probabilities on its own samples; chapter 07) back five steps, and continued with the new teacher. The fix cost about $90,000 against the estimated $2.31M, "only 4%" in the blog's words (both are Xiaomi's figures).

Who uses it and why

"Over recent months, we have observed consistent performance improvements correlating with extended RL training budget, which already exceeds 10% of the pre-training cost."

"Because most of a job’s GPU usage comes from inference, optimizing inference throughput is critical for efficient RL."

"To enable large-scale joint RL across hundreds of downstream domains, we optimized every layer of the stack — the RL framework, long-context inference, training efficiency, sandbox infrastructure, rollout scheduling, and a shared platform for environment quality checks — so that each increment of RL compute delivers stable, predictable returns."

1. DeepSeek-AI, DeepSeek-V3 Technical Report (2024), §1 Table 1
2. DeepSeek-AI, DeepSeek-R1 (2025), §2.1, App. B.4.4 Table 7
3. DeepSeek-AI, DeepSeek-V3.2 (2025), §1, §4.1
4. Microsoft AI, MAI-Thinking-1 report (2026), §1, §3.6.4
5. Xiaomi, MiMo-V2.6 Technical Report (2026), §4.1 Figure 3, §5.5 Figure 12, §6.3
6. Xiaomi, A lesson from scaling RL: the reward blind spot in optimizing for correctness (MiMo-V2.6 MOPD blog, 2026)
7. Thinking Machines, Introducing Inkling (2026), RL at scale
8. Moonshot AI, Introducing Kimi K3.1 Pro (2026)
9. Pope et al., Efficiently Scaling Transformer Inference (2022), §2
10. Pretraining notes, chapter 00: the two bills (prefill, decode, KV cache)
11. Chapter 00 (DeepSeek's posttraining compute series) and chapter 06 (partial rollouts)

12 Putting it together: two posttraining pipelines

When a lab rebuilds its posttraining from one release to the next, which parts change, and why?

Earlier chapters took one posttraining method at a time; real pipelines combine them, and each release changes several at once. This chapter follows two labs through 2025 and 2026: DeepSeek, which trains specialists and merges them, and Xiaomi's MiMo, which moved to one mixed RL run. For each, the latest pipeline comes first, end to end, then what changed across releases; a fix Xiaomi made after a release closes the chapter.

DeepSeek: from R1 to V4.1-Flash

DeepSeek-V4, end to end. After pretraining, V4 passes through three stages:

1. SFT gives the starting checkpoint (chapter 01).
2. Specialist RL. Copies of that checkpoint each train with GRPO on one domain (chapters 03 and 07); for open-ended tasks the policy itself acts as a rubric-guided judge and is trained as a judge too (chapter 04).
3. Multi-teacher on-policy distillation. One student samples its own responses and matches more than ten specialist teachers token by token (chapter 07). This stage replaced V3.2's mixed RL stage.

Alongside the stages, the expert weights and the indexer train with MXFP4 QAT and RL samples rollouts with FP4 weights (chapter 09); the finished model offers three effort modes (chapter 08). The table reads across four releases.

Figure 0.1 draws the R1, V3.2 and V4 pipelines stage by stage, and chapter 07 explains V4's switch to on-policy distillation. Three threads run across the table's rows.

Rewards. R1 scored reasoning with rules and gave a specific reason for avoiding learned reward models there:

"This decision is predicated on our observation that neural reward models are susceptible to reward hacking during large-scale reinforcement learning."

For open-ended tasks DeepSeek went the other way, to V4's self-judging policy (stage 2 above).

Where the effort goes. V3.2 spent more than a tenth of its pretraining compute on posttraining (chapter 00) and merged six specialists; V4 and V4.1-Flash merged more than ten and then more than forty teachers. V4.1-Flash is notable for what it did not change. DeepSeek kept V4's algorithms and put its effort into building tasks and environments, scaling rollouts, and running RL asynchronously so that a few very long agent trajectories no longer hold up a whole batch:

"In this release, we refrain from introducing novel post-training algorithms."

Control. Each step gave users finer control over how much the model reasons, because output tokens are a large part of serving cost. V4.1-Flash's effort level from 1 to 100 is trained by sampling several effort levels per prompt in RL, with a token penalty that depends on the effort.

MiMo: from V2-Flash to V2.6

MiMo-V2.6, end to end. MXFP4 QAT starts before posttraining, in mid-training (chapter 09). Posttraining then has three stages:

1. SFT (chapter 01).
2. One mixed RL run (MixRL): asynchronous GRPO over every task type and harness, 16 rollouts for each of 1,568 prompts per step at roughly 110K–150K tokens each, sampled from MXFP4 experts (chapters 09 and 11). Rewards are tests plus groupwise grading, which ranks the passing rollouts of a group against one another; the router stays frozen, and routing replay reuses the rollout's expert choices in training (chapter 06).
3. A second multi-teacher on-policy distillation run (MOPD2) from the RL checkpoint, with domain teachers, including ones for hard-to-verify domains, plus a repetition-specialist teacher (chapter 07).

The table sets V2.6 beside V2-Flash, its predecessor.

MiMo moved in the opposite direction from DeepSeek. V2-Flash merged separate domain teachers with multi-teacher on-policy distillation so that gains in one skill would not cause regressions in others (chapter 07); V2.6 ran one large RL job over every task type at once:

"Tasks and multiple harnesses are mixed in the same batch so capabilities reinforce each other and strategies transfer to harnesses never seen in training."

Both labs end with on-policy distillation: DeepSeek to merge many specialists, MiMo to add or fix a behavior cheaply after one mixed RL run, as the exercise below shows. MiMo also froze the expert router during RL "to maintain stable expert loads": in a test with a trainable router, the share of almost unused experts in one layer rose from 0.5% to 22% in 20 steps.

Five pressures behind the changes

Each row of the two tables answers one of five pressures. Figure 12.1 sorts the choices from chapters 03 to 12 by the pressure each one answers.

1. DeepSeek-R1 report
2. DeepSeek-V3.2 report
3. DeepSeek-V4 report
4. DeepSeek-V4.1 report
5. MiMo-V2-Flash report
6. MiMo-V2.6 report
7. MiMo blog: A lesson from scaling RL: the reward blind spot in optimizing for correctness

A Glossary

Short definitions of the terms used in these notes, with the chapter that explains each one.

Abstention

Answering "I don't know" instead of guessing. Under binary grading abstaining never pays, so models learn to guess; giving wrong answers a penalty makes abstaining the better choice when the model is unsure. ch 10

Advantage (A)

How much better a sample or token turned out than expected. Policy-gradient methods push up the probability of samples with positive advantage and push down those with negative advantage. ch 03

Agent harness (scaffold)

The program around the model that turns its outputs into tool calls, runs them, and feeds results back: which tools exist, how the context is managed, when the episode ends. The same model can score very differently in different harnesses. ch 05

Asynchronous RL

Running rollout generation and training concurrently, usually on separate GPUs, so slow, long rollouts do not leave the trainers idle. The price is that some samples come from a slightly older policy. ch 06

Baseline

A quantity subtracted from the reward before the policy-gradient update that does not depend on the sampled response, so the gradient stays unbiased while its variance falls. The prompt's expected reward is the usual choice; GRPO estimates it with the group mean. ch 03

Bradley–Terry model

The standard model of pairwise preferences: the probability that response a is preferred to b is σ(r(a) − r(b)). Reward models and DPO are both trained with it. ch 02

Brier score, log score

Scoring rules for a stated probability q of an outcome o: the Brier loss (q − o)² and the log score. Both are strictly proper: a model gets its best expected score only by reporting what it believes. Used to reward calibrated confidence. ch 04

CBRN

Chemical, biological, radiological and nuclear: the weapons domains labs filter from training data and test before releasing weights. ch 10

Chain of thought

The intermediate reasoning a model writes before its answer. Reasoning models are trained to write long ones, and effort control sets how long. ch 08

Chat template

The fixed token format that marks system, user, assistant and tool turns (and thinking blocks) in a conversation. SFT teaches the model to produce it. ch 01

Checkable (verifiable) rewards

Rewards a program can compute: comparing a final answer with a reference, running code against tests, checking a format. DeepSeek-R1 trained reasoning on them. They are still hacked through the environment around the check. ch 04

CISPO

MiniMax's policy-gradient objective (M1): instead of clipping a token's update, it caps the token's importance weight and treats it as a constant, so every token keeps a gradient. ch 03

Clip-higher

DAPO's change to PPO clipping: a larger upper clip ε_high than lower clip ε_low, so the probability of good but unlikely tokens can grow further before the gradient stops. It counters entropy collapse. ch 03

Cold start

A small SFT stage on curated long reasoning examples before reasoning RL, so RL starts from readable, well-formatted chains of thought. R1-Zero skipped it; R1 added it. ch 01

Compaction

A rewrite of an agent's context done by the harness inside an episode when the context nears its limit, for example summarizing and restarting or dropping old tool outputs. ch 05, ch 03

Context distillation

Training a model to behave as if an instruction were in its prompt: generate examples with the instruction present, delete it, and fine-tune on the result. ch 10

Critic (value function, V)

A learned model that predicts the expected reward from a given state. PPO uses it to compute advantages; GRPO replaces it with the mean reward of a group of samples. ch 03

DAPO

A set of GRPO fixes from ByteDance: clip-higher, dynamic sampling, token-level loss averaging and a soft penalty for overlong answers. ch 03

Deliberative alignment

OpenAI's safety training for gpt-oss: SFT on reasoning that cites the written safety specification (learned by context distillation), then RL with a judge that reads the specification but not the chain of thought. ch 10

DPO (direct preference optimization)

Training on preference pairs with a classification-style loss derived from the RLHF objective, with no separate reward model and no sampling during training. ch 02

Dr. GRPO

A GRPO variant that removes the division by the group's standard deviation and by response length, which otherwise bias training toward nearly-solved and nearly-hopeless prompts and toward long wrong answers. ch 03

Dynamic sampling

Dropping prompts whose samples all got the same reward (all right or all wrong), since their advantages are all zero, and sampling more prompts to fill the batch. ch 03

Effort level (e)

A setting that tells the model how much to think before answering, from a few named modes (low, high, max) to a continuous scale (DeepSeek V4.1-Flash: 1 to 100). Trained by making the length part of the reward depend on the level. ch 08

Entropy collapse

The policy becoming nearly deterministic early in RL, so its samples stop exploring and the reward stops improving. ch 03

Environment

Everything an agent interacts with during RL: a task, the tools and state it can act on (often a container with a code repository), and a verifier that scores the result. ch 05

Exposure bias

The gap between training and use in SFT: during training every prefix comes from the demonstration, but at inference it comes from the model's own samples, so one early mistake can lead the model into states it never saw. ch 01

Fail-to-pass (F2P), pass-to-pass (P2P) tests

For a software task built from a real fix: F2P tests fail before the fix and pass after it, so they check the task was solved; P2P tests pass both before and after, so they check nothing else broke. ch 05

Format reward

A small rule-based reward for following the required output format, such as putting reasoning inside thinking tags or the final answer in a box. ch 04

FP4 (MXFP4, NVFP4)

4-bit floating-point formats with a shared scale per small block of values: 32 values with a power-of-two scale for MXFP4, 16 values with an FP8 scale for NVFP4. Used to serve expert weights and KV caches cheaply. ch 09

GAE (generalized advantage estimation)

PPO's way of turning per-step value estimates into advantages, blending short- and long-horizon estimates with a parameter λ. ch 03

Generative reward model (GRM)

A language model that judges a response by writing an evaluation, often against a rubric, instead of outputting one scalar. Used for tasks that rules and tests cannot check. ch 04

Group (G)

The set of responses sampled for the same prompt in GRPO-style RL; advantages are computed relative to the group. Sizes in these notes run from 16 (DeepSeek-R1) to 128 (MAI-Thinking-1); the examples use 8. ch 03

Groupwise grading

Scoring the responses in a group by comparing them with each other rather than one at a time, for example ranking several passing solutions by quality. MiMo V2.6 uses it on top of test results. ch 12

GRPO (group relative policy optimization)

DeepSeek's critic-free version of PPO: sample a group of responses per prompt and use each one's reward minus the group mean, divided by the group's standard deviation, as its advantage. ch 03

GSPO (group sequence policy optimization)

Qwen's variant of GRPO that computes the importance ratio and clipping for a whole sequence instead of per token, which is more stable for mixture-of-experts models. ch 03

Harmony

The chat format of OpenAI's gpt-oss: ranked roles (system, developer, user, assistant, tool) and channels that separate the chain of thought, tool calls and the final answer. ch 01

IcePop (mismatch masking)

A fix for training–inference mismatch from inclusionAI's Ling team, introduced with Ring-flash-2.0 and detailed in Ring-1T: weight each token by the ratio κ_t of training-engine to inference-engine probability and drop the token when the ratio leaves a band. GLM-5 uses it in reasoning RL, and NVIDIA's Nemotron 3 Ultra in on-policy distillation. ch 06

Importance ratio (weight), ρ_t

π_θ(y_t) / π_old(y_t): how much more or less likely the current policy makes a token than the policy that sampled it. PPO-style losses clip it, and off-policy and asynchronous RL weight each token's gradient by it, usually capped, so old samples do not bias the update. ch 03, ch 06

KL penalty (β)

A term that penalizes the policy for drifting from a reference policy π_ref, measured by KL divergence. Classic RLHF relies on it; many 2025-26 reasoning recipes shrink or drop it. ch 02

KV cache, prefill, decode

The KV cache stores the keys and values of every token already read. Prefill is the pass that reads a prompt and builds the cache; decode generates one token at a time and reads the whole cache at each step, which makes rollouts memory-bound. ch 06, ch 11, pretraining ch 00

Length penalty

A reward term that lowers the reward of longer responses, measured against a cap, the other samples in the group, a reference model or a range; used to keep answers from growing without limit and to train effort levels. ch 04, ch 08

MIS-PO

StepFun's actor-critic objective (Step 3.5 Flash): tokens and trajectories whose trainer-to-engine probability ratio leaves a band are masked, and the rest are trained as on-policy, with no importance weight. ch 03

MixRL

Xiaomi's name for MiMo-V2.6's single mixed RL run over all task types and harnesses. ch 11, ch 12

Mixture of experts (MoE)

A layer with many feed-forward "experts" of which a router picks a few per token, so the model stores many parameters but uses only some for each token. Its routing causes several RL stability problems. pretraining ch 02, ch 06

MOPD (multi-teacher on-policy distillation)

Xiaomi's name for on-policy distillation from several domain teachers at once: each student sample is supervised by the teacher for its domain. Xiaomi's second run on MiMo-V2.6, MOPD2, also adds teachers for hard-to-verify domains. ch 07, ch 12

Off-policy, on-policy

On-policy training uses samples from the current model; off-policy training uses samples from another model or an older version. SFT on teacher outputs is off-policy; GRPO and on-policy distillation are (nearly) on-policy. ch 00, ch 06

On-policy distillation (OPD)

The student generates its own responses and is trained to match a teacher's token probabilities on them, for example by minimizing the reverse KL from student to teacher at every token. It gives a dense learning signal on the states the student actually visits. ch 07

Partial rollout

Pausing a very long rollout at the end of a training step and continuing it in a later step, so one long trajectory does not hold up a batch. ch 06

pass@k

The probability that at least one of k samples solves a task; pass@1 is the share solved by a single sample. DeepSeek V3.2 kept synthesized tasks only if they had nonzero pass@100. ch 00, ch 05

Policy (π_θ)

In RL, the model being trained, viewed as a probability distribution over responses given a prompt. ch 03

Post-training quantization (PTQ)

Rounding a finished model's weights to a low-precision format once, with no further training; the baseline that quantization-aware training improves on. ch 09

PPO (proximal policy optimization)

The policy-gradient method behind classic RLHF: it clips the importance ratio so each update stays close to the sampling policy, and uses a critic to estimate advantages. ch 03

Preference pair

Two responses to the same prompt with a label saying which one is better, from a human or an AI judge. ch 02

Preserved thinking

Keeping the model's earlier reasoning in the context across tool calls (and in some models across user turns) instead of deleting it after each round, so an agent does not have to re-derive its plan. ch 08

QAT (quantization-aware training)

Training with the low-precision rounding of the serving format applied in the forward pass, so the model learns weights that survive quantization. Labs now run it during posttraining to ship 4-bit experts. ch 09

Quantization-aware distillation (QAD)

NVIDIA's short repair step after post-training quantization: the quantized model is trained to match the full-precision model's outputs. ch 09

Rejection sampling

Generating several responses per prompt, keeping only those that pass a check (correct answer, judge approval), and using them as SFT data. ch 01

Return

The (discounted) sum of rewards from a token to the end of the episode; with a single reward at the end and no discount it is just that reward. ch 03

Reverse KL

KL(π_S ‖ π_T), the divergence measured under the student's own distribution. Minimizing it is mode-seeking: the student concentrates on outputs the teacher rates highly rather than covering every option. On-policy distillation minimizes it per token on the student's samples. ch 07

Reward hacking

The policy raising its reward by exploiting flaws in the reward (a lenient judge, a weak test, a format loophole) instead of doing the task. ch 04

Reward model

A model trained on preference pairs to output a scalar score for a response; RLHF optimizes the policy against it. ch 02

RLHF (RL from human feedback)

The 2022-24 recipe: train a reward model on human preference pairs, then optimize the policy against it with PPO and a KL penalty. ch 02

Rollout

One complete sampled response, or one complete agent episode with all its tool calls and results, generated during RL. ch 00, ch 11

Routing replay

Recording which experts each token used during rollout and reusing those choices in the training forward pass, so the two engines compute the probabilities of the same network path. DeepSeek calls it Keep Routing; Xiaomi and Peking University published it as Rollout Routing Replay (R3). ch 06

Rubric

A prompt-specific list of criteria a judge uses to score a response, written by people or generated from good and bad examples. ch 04

Sandbox

An isolated container in which an agent's commands run during RL or evaluation, so it can install packages, edit files and run tests safely. ch 05

SAO

GLM's name (from the GLM-5.3 blog) for its return to critic-based policy optimization in agentic RL, training every sub-trace of a rollout as a trajectory with a token-level loss. ch 03

Score-function (log-derivative) identity

∇π_θ(y) = π_θ(y) ∇log π_θ(y), which turns the gradient of an expected reward into an expectation that can be estimated from samples. ch 00

SFT (supervised fine-tuning)

Next-token training on curated prompt and response pairs, with the loss applied only to the response tokens. ch 01

Specialist (teacher) model

A copy of the model trained further on one domain, such as math, code or agentic search, whose skills are later merged into a single model. ch 07

Staleness

How many policy updates old the sampler was when it produced a rollout. Asynchronous systems cap it; MiMo V2.6, for example, trains "at a staleness of 4". ch 06

STaR (self-taught reasoner)

A loop that generates rationales, keeps those reaching the correct answer, fine-tunes on them and repeats; an early form of learning from the model's own checked samples. ch 01

Straight-through estimator (STE)

Back-propagating through a rounding step as if it were the identity, so quantization-aware training can update full-precision master weights. ch 09

Training–inference mismatch

The difference between token probabilities computed by the fast rollout engine and by the training engine for the same weights, caused by different kernels, precision or expert choices. Left alone it biases the gradient. ch 06

Trust region

A limit on how far one update may move the policy from the one that sampled the data; PPO approximates it by clipping the importance ratio. ch 03

Verifier

The program or model that decides whether a response succeeded: an answer checker, a test suite, a rubric judge. ch 04

Weight merging

Averaging the parameters of several models fine-tuned from the same base, with no further training. Kimi k1.5 merged a long-chain-of-thought model and a short one this way. ch 07

B Citation

To give credit if you find this work useful, please cite:

@misc{ma2026openrecipesposttrain,
  title={Reading Open-Weight Recipes, Part 3: What 2025--26 Model Reports Disclose About Posttraining},
  author={Ma, Ziqiao},
  howpublished={\url{https://mars-tin.github.io/blogs/posts/open_recipes_posttraining}},
  note={Blog post},
  year={2026}
}