| DeepSeek | V3 | Pre-Training | pretrain | 14.8T tokens at 4K context, including the LR decay tail (cosine 2.2e-4 -> 2.2e-5 over 4.3T, then 333B @2.2e-5 and 167B @7.3e-6). Pretrain by the rule: the decay sits inside the main schedule and the lab does not name it as a separate phase. ↗ |
| DeepSeek | V3 | Long Context Extension | midtrain | YaRN 4K -> 32K -> 128K, two phases of 1000 steps each (batch 1920 then 480), LR 7.3e-6; 119K GPU hours ↗ |
| DeepSeek | V3 | Supervised Fine-Tuning | posttrain | 1.5M instances, 2 epochs; reasoning data distilled from internal R1 via expert models + rejection sampling ↗ |
| DeepSeek | V3 | Reinforcement Learning | posttrain | GRPO with rule-based + model-based RMs over coding, math, writing, role-play, QA prompts ↗ |
| DeepSeek | V3 | Expert models for R1 data generation (SFT+RL experts, rejection sampling) | posttrain | Domain experts trained with SFT+RL generate the reasoning SFT data via rejection sampling; this is distillation from R1 into V3 ↗ |
| DeepSeek | R1 | DeepSeek-R1-Zero (pure RL on V3-Base) | posttrain | GRPO for 10,400 steps on DeepSeek-V3-Base with rule-based accuracy + format rewards, no SFT ↗ |
| DeepSeek | R1 | Cold start (long-CoT SFT) -> R1-Dev1 | posttrain | Thousands of curated long-CoT samples; SFT of V3-Base for 2-3 epochs ↗ |
| DeepSeek | R1 | First RL stage (reasoning RL) -> R1-Dev2 | posttrain | GRPO with rule-based + language-consistency rewards, clip epsilon 10, max 32,768 tokens ↗ |
| DeepSeek | R1 | Rejection sampling + second SFT -> R1-Dev3 | posttrain | ~600K reasoning + ~200K non-reasoning samples; fine-tunes DeepSeek-V3-Base again (per App. B.4.2) ↗ |
| DeepSeek | R1 | Second RL stage (all scenarios) -> R1 | posttrain | 1,700 steps at temp 0.7; rule rewards for reasoning, helpful/safety RMs for general data (last 400 steps only) ↗ |
| DeepSeek | R1 | DeepSeek-R1 distillation (R1-Distill series) | posttrain | SFT-only on 800K R1-generated samples into Qwen2.5 / Llama-3 dense bases ↗ |
| DeepSeek | Prover-V2 | stage 1: expert iteration (non-CoT) with subgoal curriculum | posttrain | Iterative SFT on Lean-verified proofs of hard problems ↗ |
| DeepSeek | Prover-V2 | stage 2: cold-start CoT SFT + reasoning RL | posttrain | SFT on V3-Base-671B with cold-start CoT data, then GRPO with binary Lean reward ↗ |
| DeepSeek | Prover-V2 | 7B distillation: context extension 4K->32K | midtrain | Prover-V1.5-Base-7B context extended before fine-tuning on 671B rollouts (the fine-tune and RL are posttrain); midtrain as continued training of an earlier base ↗ |
| DeepSeek | R1-0528 | post-training upgrade | posttrain | Card states only more compute and algorithmic optimizations during post-training ↗ |
| DeepSeek | V3.1 | two-phase long context extension (V3.1-Base) | midtrain | Original V3 base checkpoint continued at 32K (630B tokens) then 128K (209B tokens) with extra long documents ↗ |
| DeepSeek | V3.1 | post-training on V3.1-Base | posttrain | Hybrid thinking/non-thinking model and tool-use optimization; method not described ↗ |
| DeepSeek | V3.2-Exp | Continued Pre-Training: Dense Warm-up Stage | midtrain | Indexer-only training with dense attention on 128K data, 1000 steps, 2.1B tokens, LR 1e-3; continued training of an existing base to add an attention component ↗ |
| DeepSeek | V3.2-Exp | Continued Pre-Training: Sparse Training Stage | midtrain | All parameters adapted to top-2048 sparse attention, 15000 steps x 480 x 128K = 943.7B tokens, LR 7.3e-6, data = V3.1-Terminus 128K extension mix ↗ |
| DeepSeek | V3.2-Exp | Post-Training: Specialist Distillation + Mixed RL Training | posttrain | SFT on data distilled from RL-trained domain specialists, then one merged GRPO stage; identical to V3.1-Terminus post-training ↗ |
| DeepSeek | Math-V2 | iterative verifier/generator RL with rejection fine-tuning | posttrain | All Math-V2 training is post-training on V3.2-Exp-Base/SFT ↗ |
| DeepSeek | V3.2 | Continued Pre-Training (dense warm-up + sparse training) | midtrain | Same DSA continued pretraining as V3.2-Exp (card base_model = V3.2-Exp-Base) ↗ |
| DeepSeek | V3.2 | Specialist Distillation | posttrain | Six-domain specialists (each RL-trained) generate SFT data, thinking and non-thinking ↗ |
| DeepSeek | V3.2 | Cold-Start (thinking in tool-use) | posttrain | Prompt-engineered system prompts produce reasoning-with-tool-call trajectories that seed RL ↗ |
| DeepSeek | V3.2 | Mixed RL Training (scaled GRPO) | posttrain | Single RL stage (reasoning + agent + human alignment), >10% of pretraining compute, thousands of steps; Speciale = reasoning-only RL with reduced length penalty + Math-V2 data/reward ↗ |
| DeepSeek | V4 | Pre-Training (main phase, constant LR after 2000-step warmup) | pretrain | Flash 32T / Pro 33T tokens with Muon; dense attention warmup (first 1T tokens for Flash) then CSA/HCA sparse attention for most of training ↗ |
| DeepSeek | V4 | sequence-length extension inside pre-training (4K -> 16K -> 64K -> 1M) | pretrain | Context grows in steps during the same run; per-stage token counts not given. Pretrain: length growth the lab does not name as a separate phase (it calls all of this pre-training). ↗ |
| DeepSeek | V4 | introduction of sparse attention (indexer warmup, then sparse training) | pretrain | Architecture switch at 64K sequence length; sparse attention then covers most of pre-training, so kept in pretrain ↗ |
| DeepSeek | V4 | mid-training phase (agentic data) | midtrain | Agentic data added to strengthen coding; no further detail ↗ |
| DeepSeek | V4 | learning-rate decay near end of pre-training | pretrain | Cosine decay to 10% of peak; MTP loss weight cut from 0.3 to 0.1 at decay start. Pretrain: the decay sits inside the main schedule and is not named as a separate phase. ↗ |
| DeepSeek | V4 | Specialist Training (SFT + GRPO RL per domain and per reasoning-effort mode) | posttrain | Each domain expert: SFT on domain data then GRPO with domain rewards / GRM ↗ |
| DeepSeek | V4 | On-Policy Distillation (multi-teacher) | posttrain | >10 teachers distilled into one student by full-vocabulary reverse-KL OPD; replaces V3.2's mixed RL stage ↗ |
| DeepSeek | V4 | FP4 Quantization-Aware Training (during post-training) | posttrain | MXFP4 QAT for MoE experts and indexer QK path, applied to student, teachers and reference models ↗ |
| DeepSeek | V4-DSpark | DSpark speculative-decoding module attached to the V4-Pro checkpoint | posttrain | Separate draft module trained against target outputs; target weights unchanged ↗ |
| DeepSeek | V4-Flash-Vision-Exp | incorporating visual modules and continued training | midtrain | Continued training of the V4-Flash LM with added vision encoder/aligner (adding vision to an existing LM) ↗ |
| DeepSeek | V4.1-Flash | Vision encoder training (contrastive pre-training + autoregressive fine-tuning) | pretrain | DeepSeek-ViT trained before LM pretraining: SigLIP loss on ~47B pairs, then AR fine-tuning with a 4B MoE LLM on 236B tokens; component pretraining ↗ |
| DeepSeek | V4.1-Flash | Pre-Training (45T multimodal tokens, constant LR to 28T) | pretrain | From scratch, 64K sequence length, sparse attention throughout, LR 2.6e-4 after 2000-step warmup ↗ |
| DeepSeek | V4.1-Flash | Sequence-length extension to 1M (at 34T tokens, inside pretraining) | midtrain | 64K -> 1M at 34T tokens; remaining ~11T tokens at 1M. Midtrain: it runs inside the learning-rate decay stage the lab names, which starts at 28T. ↗ |
| DeepSeek | V4.1-Flash | Learning-rate decay stage (28T-40T) + constant low LR (40T-45T) | midtrain | Cosine decay 2.6e-4 -> 2.6e-5, then flat; vision encoder unfrozen at decay onset with smaller LR. Midtrain: the lab names it a learning-rate decay stage; with the 1M extension it covers 28T-45T, 17T of 45T. ↗ |
| DeepSeek | V4.1-Flash | DSpark dedicated stage after pre-training (backbone frozen) | posttrain | Trains only the speculative-decoding drafter; mapped to posttrain because it follows pretraining and does not change backbone weights ↗ |
| DeepSeek | V4.1-Flash | Supervised fine-tuning (SFT) | posttrain | First post-training stage; data not described ↗ |
| DeepSeek | V4.1-Flash | Reinforcement learning in synthesized tasks | posttrain | Asynchronous GRPO-style RL across text, multimodal and agentic tasks, multi-scaffold, effort-conditioned; includes FP4 main-KV QAT and Hierarchical Sparse Indexer adaptation; successive runs re-initialized from merged c… ↗ |
| DeepSeek | V4.1-Flash | Large-scale on-policy distillation (last stage) | posttrain | Full-vocabulary OPD from >40 heterogeneous teachers on all domains ↗ |
| Kimi | Moonlight | stage 1 + stage 2 (0-5.2T tokens) | pretrain | 2k-step warmup to 4.2e-4, cosine decay to 4.2e-5 by 5.2T tokens, 8K context, batch 2048->4096 examples ↗ |
| Kimi | Moonlight | stage 3 / cooldown stage (5.2T-5.7T tokens) | midtrain | LR re-warm to 1e-4 then linear decay to 0 over 500B tokens; highest-quality math/code/reasoning data ↗ |
| Kimi | Kimi-VL | ViT Training Stages (CoCa-style + alignment) | midtrain | Vision encoder continual pretraining from SigLIP-SO-400M (2T) then 0.1T alignment to the LM updating only MoonViT and projector ↗ |
| Kimi | Kimi-VL | Joint Pre-training Stage | midtrain | Continued pretraining of the Moonlight 5.2T checkpoint with text + rising multimodal share, 1.4T tokens, same LR scheduler ↗ |
| Kimi | Kimi-VL | Joint Cooldown Stage | midtrain | 0.6T high-quality text + multimodal data with synthetic QA; LR re-warmed ↗ |
| Kimi | Kimi-VL | Joint Long-context Activation Stage | midtrain | 0.3T, 8K->32K->128K, RoPE base 50K->800K, 25% long / 75% replay ↗ |
| Kimi | Kimi-VL | Joint SFT; Long-CoT SFT; RL | posttrain | Joint SFT at 32K then 128K gives Kimi-VL (Instruct); long-CoT SFT + online mirror-descent RL gives Kimi-VL-Thinking ↗ |
| Kimi | Kimi-Linear | Pre-training recipe | pretrain | 4K context, MuonClip, WSD, LR 1.1e-3, 32M-token batch, 5.7T tokens from the K2 corpus ↗ |
| Kimi | Kimi-Linear | annealing schedule and long-context activation phase (as in K2) | midtrain | K2's anneal + long-context activation reused; final checkpoint supports 1M context ↗ |
| Kimi | Kimi-Linear | SFT recipe; RL recipe | posttrain | Multi-stage SFT on extended K2 SFT data; K1.5 RL algorithm with truncated IS, adaptive KL/mini-batch, PTX loss ↗ |
| Kimi | K2 | pre-training (WSD stable phase) | pretrain | 10T tokens at constant LR 2e-4 after 500-step warm-up, 4k context, MuonClip, 67M-token batch ↗ |
| Kimi | K2 | pre-training (WSD cosine decay) | pretrain | 5.5T tokens with cosine LR decay 2e-4 -> 2e-5. Pretrain: the decay is inside the main WSD schedule of the 15.5T run, the lab calls it pre-training and states no data-mix change; only the annealing and long-context phases that follow are midtrain. ↗ |
| Kimi | K2 | annealing phase + long-context activation | midtrain | LR 2e-5 -> 7e-6; 400B tokens at 4k then 60B tokens at 32k; YaRN to 128k ↗ |
| Kimi | K2 | supervised fine-tuning (incl. agentic data synthesis) | posttrain | Muon SFT on multi-domain instruction data generated by K1.5/expert models plus synthetic tool-use trajectories ↗ |
| Kimi | K2 | joint reinforcement learning | posttrain | Single joint RL stage over verifiable-reward gym and self-critique rubric reward ↗ |
| Kimi | K2-Thinking | post-training QAT | posttrain | INT4 weight-only QAT on MoE weights during post-training ↗ |
| Kimi | K2.5 | Kimi K2 base model (foundation) | pretrain | Inherited: K2 1.04T-A32B MoE pretrained on 15T text tokens with MuonClip/QK-Clip; K2.5 starts from a near-end K2 checkpoint ↗ |
| Kimi | K2.5 | ViT Training stage | midtrain | MoonViT-3D continual pretraining from SigLIP with caption loss, ~1T tokens aligned to Moonlight-16B-A3B, then projector-only bridging to the 1T LLM; part of adding vision to an existing LM ↗ |
| Kimi | K2.5 | Joint Pre-training | midtrain | Lab calls it pre-training, but it continues a near-end K2 checkpoint with 15T vision-text tokens at 4K (adds vision to an existing LM, changes mix) -> my midtrain ↗ |
| Kimi | K2.5 | Joint Long-context Mid-training | midtrain | 500B->200B tokens, 32,768 -> 262,144 via YaRN, high-quality text & multimodal, long text/video, reasoning, long-CoT ↗ |
| Kimi | K2.5 | Zero-vision SFT | posttrain | Text-only SFT on responses from K2 / K2 Thinking / in-house experts ↗ |
| Kimi | K2.5 | Joint multimodal RL (incl. outcome-based visual RL, RFT, PARL) | posttrain | Ability-organized RL on text+vision, GRMs, Toggle; PARL trains the swarm orchestrator ↗ |
| Kimi | K2.5 | Native INT4 quantization | posttrain | Same native INT4 method as K2-Thinking (QAT in post-training per K2-Thinking card) ↗ |
| Kimi | K3 | Pre-Training (native multimodal, 8K then 64K) | pretrain | Text and vision jointly optimized from step 0; Per-Head Muon, cosine LR, 8K context extended to 64K inside the main run ↗ |
| Kimi | K3 | Long-Context Extension / cooldown phase | midtrain | 256K -> 1M context during cooldown with upsampled long documents/videos and synthetic long-range tasks; midtrain because the lab names it a cooldown. The 8K->64K step inside the main run is length growth the lab does not name as a phase, so it stays in pretrain, as DeepSeek V4's does. ↗ |
| Kimi | K3 | Supervised Fine-Tuning (cold start) | posttrain | Cold-start policy for RL; agentic trajectories from prior Kimi specialists; QAT starts here ↗ |
| Kimi | K3 | Reinforcement Learning (domain x effort experts, incl. Reasoning Effort RL) | posttrain | 3 domains x {low, high, max} = 9 experts; partial rollouts; token-budget reward control ↗ |
| Kimi | K3 | Multi-Teacher On-Policy Distillation (MOPD) | posttrain | 9 experts consolidated into one model with per-token on-policy distillation reward ↗ |
| Kimi | K3 | Deployment-Aware Post-Training: MXFP4 QAT | posttrain | MXFP4 expert weights / MXFP8 activations through SFT and RL ↗ |
| Kimi | K3 | Draft Model Fine-Tuning | posttrain | MTP layer tuned into an EAGLE-3 draft with LK loss, target frozen (inference aid, not a capability stage) ↗ |
| GLM | GLM-4.5 | Pre-training stage 1 (general corpus, 15T, 4K) | pretrain | Mainly web documents at 4K; first part of one cosine LR run ↗ |
| GLM | GLM-4.5 | Pre-training stage 2 (code & reasoning corpus, 7T, 4K) | pretrain | Pretrain: data re-weighting during the main run (7T of 22T, toward code/math/science) at the same 4K length and on the same cosine LR, before the main decay; the lab calls it pre-training. MiMo-7B's stage 2 differs because it comes after the main decay. ↗ |
| GLM | GLM-4.5 | Mid-training: repo-level code training | midtrain | 4K -> 32K, concatenated repo files + issues/PRs/commits, 500B tokens, RoPE base 10K -> 1M ↗ |
| GLM | GLM-4.5 | Mid-training: synthetic reasoning data training | midtrain | 32K, synthesized reasoning traces for math/science/code competitions, 500B tokens ↗ |
| GLM | GLM-4.5 | Mid-training: long-context & agent training | midtrain | 32K -> 128K, up-sampled long docs + synthetic agent trajectories, 100B tokens; LR decays to 2.5e-5 by end ↗ |
| GLM | GLM-4.5 | Post-training stage 1: Expert Training | posttrain | Cold-start SFT + RL for Reasoning, Agent and General-chat experts ↗ |
| GLM | GLM-4.5 | Post-training stage 2: Unified Training | posttrain | Overall SFT distilling the experts (hybrid thinking) followed by general RL ↗ |
| GLM | GLM-5 | Pre-training (27T) | pretrain | 4K-context pre-training prioritizing code and reasoning early ↗ |
| GLM | GLM-5 | Mid-training (32K / 128K / 200K) | midtrain | 1T at 32K, 500B at 128K, 50B at 200K; long docs and agent trajectories up-sampled; SWE issue-PR data ↗ |
| GLM | GLM-5 | Continued pre-training with DSA (dense warm-up + sparse adaptation) | midtrain | Architecture change (MLA -> MLA+DSA) via continued training from the end-of-mid-training base: 1,000-step indexer warm-up, then 20B tokens of sparse adaptation on mid-training data ↗ |
| GLM | GLM-5 | SFT (incl. INT4 QAT) | posttrain | Multi-task SFT up to 202,752 tokens with interleaved/preserved thinking; INT4 QAT applied here ↗ |
| GLM | GLM-5 | Reasoning RL -> Agentic RL -> General RL | posttrain | Sequential RL stages ↗ |
| GLM | GLM-5 | On-Policy Cross-Stage Distillation | posttrain | Final stage; earlier-stage checkpoints as teachers ↗ |
| GLM | GLM-4.5V | Multimodal pre-training | midtrain | Continued pre-training of an existing LLM (GLM-4.5-Air; GLM-4-9B-0414 for 4.1V) with a ViT and image-text data: 120,000 steps, seq 8,192, batch 1,536 ↗ |
| GLM | GLM-4.5V | Long-context continual training | midtrain | Seq 32,768 with video and long interleaved data, 10,000 steps ↗ |
| GLM | GLM-4.5V | SFT (long-CoT cold start) + RL (RLCS) | posttrain | Cold-start SFT then GRPO-based multi-domain RL ↗ |
| GLM | GLM-4.6V | 128K context extension stage | midtrain | Extra 2,000 steps at 131,072 tokens, global batch 128 ↗ |
| GLM | GLM-Z1-0414 | Cold start + extended RL + general RL from pairwise ranking feedback | posttrain | Built on GLM-4-32B-0414 ↗ |
| GLM | GLM-5.2 | Mid-training with IndexShare at 128K | midtrain | IndexShare architecture trained from mid-training onward; base model not named ↗ |
| GLM | GLM-5.2 | Parallel OPD merging >10 experts | posttrain | Expert models merged by on-policy distillation in ~2 days ↗ |
| GLM | GLM-5.3 | Scaled post-training on the GLM-5.2 base | posttrain | No new pre-/mid-training ↗ |
| GLM | GLM-5.3-Flash | Pre-training of a newly trained base (30T multimodal tokens) | pretrain | New architecture trained from scratch ↗ |
| MiMo | MiMo-7B | Pre-training Stage 1 | pretrain | All sources except synthetic reasoning responses; 8K context; includes warmup, 10.2T constant LR and a 7.5T cosine decay to 3e-5 (~17.8T tokens) ↗ |
| MiMo | MiMo-7B | Pre-training Stage 2 | midtrain | Math+code upsampled to ~70% at 8K, LR held at 3e-5 (4T tokens). Midtrain: a distinct stage after the main decay that changes the mix, although the lab calls it pre-training. ↗ |
| MiMo | MiMo-7B | Pre-training Stage 3 | midtrain | +~10% synthetic reasoning responses, 8K->32K context, RoPE base 640K, final LR decay to 1e-5 (~2T tokens) ↗ |
| MiMo | MiMo-7B | MTP layer fine-tuning | posttrain | Two extra MTP layers fine-tuned after pretraining with the main model frozen (inference speedup only) ↗ |
| MiMo | MiMo-7B | SFT / RL (RL-Zero from Base, RL from SFT) | posttrain | ~500K-sample SFT then modified-GRPO RL; RL-Zero applies RL directly to the base ↗ |
| MiMo | MiMo-VL-7B | Stages 1-3 (projector warmup, VL alignment, multimodal pre-training) | midtrain | Continued pretraining of MiMo-7B-Base with vision (adding a ViT + projector to an existing LM), 8K context, ~1.87T tokens ↗ |
| MiMo | MiMo-VL-7B | Stage 4 'long-context SFT' (lab counts it as pre-training) | midtrain | 8K->32K context with long documents/videos and much more long-form reasoning data, 550B tokens; a long-context + reasoning-upsampling phase despite the SFT name ↗ |
| MiMo | MiMo-VL-7B | Post-training: MORL | posttrain | On-policy GRPO mixing RLVR and RLHF ↗ |
| MiMo | V2-Flash | Stage 1 (Pre-training, 0-22T) | pretrain | General corpus at native 32K; includes a 10T cosine decay to 1e-4 at the end ↗ |
| MiMo | V2-Flash | Stage 2 (Mid-training, 22-26T) | midtrain | Code-centric upsampling + ~5% synthetic reasoning; LR 1e-4 -> 3e-5 ↗ |
| MiMo | V2-Flash | Stage 3 (Context Extension, 26-27T) | midtrain | 32K -> 256K, GA RoPE base 5M, long-range data upsampled ↗ |
| MiMo | V2-Flash | Post-training Stage 1: SFT | posttrain | Millions of samples, thinking + non-thinking ↗ |
| MiMo | V2-Flash | Post-training Stage 2: Domain-Specialized Training | posttrain | Independent RL (or SFT) per domain to build teachers: search, coding, tool use, math, general reasoning, safety ↗ |
| MiMo | V2-Flash | Post-training Stage 3: MOPD | posttrain | On-policy multi-teacher distillation + ORM rewards into one student ↗ |
| MiMo | V2.5 | 1. Text Pre-training | pretrain | Text-only LLM backbone pretraining ↗ |
| MiMo | V2.5 | 2. Projector Warmup + 3. Multimodal Pre-training | midtrain | Continued pretraining that adds in-house vision/audio encoders to the text backbone ↗ |
| MiMo | V2.5 | 4. SFT & Agentic Post Training | posttrain | SFT on agentic data; straddles midtrain because the 32K -> 256K -> 1M context extension happens here ↗ |
| MiMo | V2.5 | 5. RL & MOPD Training | posttrain | RL plus multi-teacher on-policy distillation ↗ |
| MiMo | V2.5-Pro | Pre-training (27T, native 32K, extended to 1M) | pretrain | Text pretraining; where the 1M extension happens is not stated ↗ |
| MiMo | V2.5-Pro | SFT -> Domain-Specialized Training -> MOPD | posttrain | Same three-stage post-training as V2-Flash ↗ |
| MiMo | V2.6 | Pre-training stage 1 (text-only) | pretrain | Flash 26T / Pro 27T text tokens; AdamW; 32K extended to 256K partway, which stays in pretrain as length growth the lab does not name as a phase ↗ |
| MiMo | V2.6 | Pre-training stage 2 (omni) | midtrain | Continued pretraining that attaches the in-house ViT and audio encoder (Flash 22T, Pro 3T); the lab calls it pre-training ↗ |
| MiMo | V2.6 | Mid-training (agent-centric) | midtrain | 256K then 1M; Muown optimizer; MXFP4 QAT; reward-hacking self-correction data ↗ |
| MiMo | V2.6 | Short SFT | posttrain | Brief SFT before RL ↗ |
| MiMo | V2.6 | Mixed RL ('You Only RL Once') | posttrain | Single asynchronous GRPO run over code/general/visual/cyber with multiple harnesses, ~30 steps ↗ |
| MiMo | V2.6 | MOPD2 | posttrain | Multi-prefix multi-teacher on-policy distillation after mixed RL ↗ |
| MiMo | V2.6-MOPD | MOPD (rolled back 5 steps) + repetition-specialist teacher | posttrain | Short specialist RL teacher then teacher-prefix OPD ↗ |
| MiMo | V2.6-Distill-Qwen-9B | SFT on MiMo-generated data; domain GRPO | posttrain | Distillation SFT then per-domain GRPO baselines on released environments ↗ |
| Gemma | Gemma-3 | Pre-training (with knowledge distillation) | pretrain | KD from an unnamed teacher (256 sampled logits/token), 2T-14T tokens of text+images at 32K sequence length ↗ |
| Gemma | Gemma-3 | Long-context scaling at the end of pre-training | pretrain | 4B/12B/27B scaled from 32K to 128K with RoPE rescaling (factor 8). Pretrain: length growth at the end of the main run that the lab does not name as a separate phase. ↗ |
| Gemma | Gemma-3 | Instruction-Tuning (distillation + RL finetuning) | posttrain | KD from a large IT teacher, then RL with improved BOND/WARM/WARP using WARM reward models, code execution and math ground truth ↗ |
| Gemma | Gemma-3 | Quantization Aware Training | posttrain | ~5,000-step QAT finetune producing int4 / SFP8 checkpoints ↗ |
| Gemma | Gemma-3n | MatFormer training | pretrain | E2B sub-model trained jointly inside E4B; post-training not described for 3n ↗ |
| Gemma | Gemma-4 | Pre-training (similar to Gemma 3) | pretrain | Text+image (+audio for E2B/E4B/12B) pre-training with frozen encoders; tokens, distillation and context schedule not disclosed ↗ |
| Gemma | Gemma-4 | Instruction Tuning (similar to Gemma 3, plus thinking mode) | posttrain | Method details not given beyond data filtering and the added thinking mode ↗ |
| Gemma | Gemma-4 | Quantization-Aware Training | posttrain | QAT for mobile (int2/int4 + int8 act) and Q4_0 formats, and for the vision/audio encoders ↗ |
| Gemma | Gemma-4 | Multi-Token Prediction drafter training | posttrain | Auxiliary drafter head trained 'with our models'; when it is trained relative to pre-training is not stated, mapped to posttrain as an add-on ↗ |
| Gemma | Gemma-4-12B | Pre-training from scratch (encoder-free) | pretrain | Trained from scratch with projection modules instead of encoders; post-training shared with the Gemma 4 recipe ↗ |
| Microsoft AI | MAI-DS-R1 | post-training (unspecified method) | posttrain | DeepSeek-R1 post-trained on ~460k safety / bias examples ↗ |
| Microsoft AI | MAI-Thinking-1 | Pre-training | pretrain | 30T tokens at 16,384 context, 8,192 GB200 ↗ |
| Microsoft AI | MAI-Thinking-1 | Mid-training phase 1 | midtrain | 3.4T tokens at 65,536 context; STEM/math 35%, code 55%; cosine LR 2e-5 -> 1e-5 ↗ |
| Microsoft AI | MAI-Thinking-1 | Mid-training phase 2 (context extension) | midtrain | 150B tokens at 262,144 context, constant LR 1e-6, same mix re-packed ↗ |
| Microsoft AI | MAI-Thinking-1 | STEM Climb / Agentic Climb / Helpfulness and Safety Climb | posttrain | Three specialist GRPO-style RL runs from the mid-trained model, no SFT cold start ↗ |
| Microsoft AI | MAI-Thinking-1 | Self-distillation | posttrain | SFT of a mid-trained checkpoint on the model's own RL rollouts, used between RL climb segments ↗ |
| Microsoft AI | MAI-Thinking-1 | Trace Distillation SFT (consolidation SFT) | posttrain | Three specialist teachers distilled into one model, 4 epochs ↗ |
| Microsoft AI | MAI-Thinking-1 | Final Climb (consolidation RL) | posttrain | Lightweight RL for safety, over-refusal and style at 128k ↗ |
| Microsoft AI | MAI-Base-2 | Pre-training (incl. native vision data) | pretrain | 50T tokens, 16k GB200, 38 days; vision data trained jointly from the start ↗ |
| Thinking Machines | Inkling | pre-training | pretrain | 45T tokens of text, images, audio and video; Muon+Adam; WD tied to LR^2 ↗ |
| Thinking Machines | Inkling | (none described) | midtrain | No annealing, mid-training or context-extension phase described, although the model supports 1M tokens ↗ |
| Thinking Machines | Inkling | initial SFT (bootstrap) | posttrain | SFT on synthetic data from open-weights models incl. Kimi K2.5; small share of compute ↗ |
| Thinking Machines | Inkling | large-scale asynchronous RL | posttrain | >30M rollouts over two long continuous runs; includes effort control, calibration, rubric/claims graders, safety ↗ |
| Thinking Machines | Inkling-Small | pre-training (changed data mix and recipe) | pretrain | Separate pretraining run started after Inkling's; details not disclosed ↗ |
| Thinking Machines | Inkling-Small | on-policy distillation from Inkling (preview checkpoint) | posttrain | Inkling as teacher, used 'in part' to post-train Inkling-Small (preview) ↗ |
| Thinking Machines | Inkling-Small | continued agentic coding RL | posttrain | Two more weeks of agentic-coding RL from the preview checkpoint to the released model ↗ |
| Qwen | Qwen2.5-VL | ViT trained from scratch (CLIP pre-training) | midtrain | New window-attention ViT trained from scratch (CLIP, then alignment) to attach to the finished Qwen2.5 LLM. Midtrain by rule (3): adding vision to an existing LM. ↗ |
| Qwen | Qwen2.5-VL | Visual Pre-Training | midtrain | 1.5T tokens at 8,192 (captions, visual knowledge, OCR), ViT only. Midtrain by rule (3): continued training around the Qwen2.5 base. ↗ |
| Qwen | Qwen2.5-VL | Multimodal Pre-Training | midtrain | 2T tokens at 8,192, all parameters: interleaved, VQA, math, grounding, video, agent and pure-text data. Midtrain by rule (3). ↗ |
| Qwen | Qwen2.5-VL | Long-Context Pre-Training | midtrain | 0.6T tokens at 32,768 with long video, long agent and long document data. Midtrain: context extension. ↗ |
| Qwen | Qwen2.5-VL | Supervised Fine-Tuning (SFT) | posttrain | ~2M ChatML samples, half text; domain-tailored rule and reward-model filtering; rejection-sampled CoT; ViT frozen. ↗ |
| Qwen | Qwen2.5-VL | Direct Preference Optimization (DPO) | posttrain | Single-pass DPO on image-text and text preference data, ViT frozen. ↗ |
| Qwen | Qwen2.5-1M | Stage 1 (4K, intermediate Qwen2.5 base) | pretrain | Reuses an intermediate Qwen2.5 Base checkpoint trained at 4,096 tokens. Pretrain: this is Qwen2.5's own from-scratch run, not new training for this release. ↗ |
| Qwen | Qwen2.5-1M | Stage 2 (32K with ABF) | midtrain | Context 4K -> 32K with RoPE base 10,000 -> 1,000,000 (ABF), as in other Qwen2.5 models. Midtrain: a context-extension phase. ↗ |
| Qwen | Qwen2.5-1M | Stages 3-5 (64K, 128K, 256K) | midtrain | 64K / 128K / 256K with RoPE base 1M / 5M / 10M; 75% max-length + 25% shorter sequences; natural long text plus synthetic long-range tasks. Midtrain: named context extension. ↗ |
| Qwen | Qwen2.5-1M | Two-stage Supervised Fine-tuning | posttrain | Short instructions (<=32K) first, then a mix of short and long (to 256K) built from Qwen-Agent-generated long-document QA. ↗ |
| Qwen | Qwen2.5-1M | Reinforcement Learning (offline, DPO-like) | posttrain | Offline preference optimization on short (<=8,192-token) pairs reused from Qwen2.5; gains transfer to long context (LongBench-Chat 7B: 7.32 -> 8.08). ↗ |
| Qwen | QwQ-32B | Cold-start checkpoint | posttrain | Cold-start SFT of Qwen2.5-32B before RL; data not described. ↗ |
| Qwen | QwQ-32B | RL stage 1 (math and coding) | posttrain | Outcome-reward RL with a math accuracy verifier and a code-execution server instead of reward models; steady gains as episodes grow. ↗ |
| Qwen | QwQ-32B | RL stage 2 (general capabilities) | posttrain | Short RL run with a general reward model and rule-based verifiers for instruction following, preference and agent skills. ↗ |
| Qwen | Qwen2.5-VL-32B | Reinforcement learning optimization | posttrain | RL on top of the Qwen2.5-VL recipe targeting answer style, math reasoning and fine-grained image understanding; algorithm and data not given. ↗ |
| Qwen | Qwen2.5-Omni | Pre-training stage 1 (encoders, LLM locked) | midtrain | LLM from Qwen2.5 frozen; Qwen2.5-VL vision and Whisper-large-v3 audio encoders trained (adapters first). Midtrain by rule (3). ↗ |
| Qwen | Qwen2.5-Omni | Pre-training stage 2 (all parameters) | midtrain | +800B image/video, 300B audio, 100B video-with-audio tokens plus text, at 8,192. Midtrain by rule (3). ↗ |
| Qwen | Qwen2.5-Omni | Pre-training stage 3 (32K) | midtrain | Long audio and video added; all data extended to 32,768 tokens. Midtrain: context extension. ↗ |
| Qwen | Qwen2.5-Omni | Thinker instruction fine-tuning | posttrain | SFT on text, visual, audio and mixed-modality dialogues. ↗ |
| Qwen | Qwen2.5-Omni | Talker: ICL, DPO, speaker fine-tuning | posttrain | Speech continuation, DPO ranked by WER and pause errors, then multi-speaker fine-tuning. ↗ |
| Qwen | Qwen3 | General Stage (S1) | pretrain | >30T tokens at 4,096 in 119 languages; per-model LR schedule and batch size from in-house scaling laws. ↗ |
| Qwen | Qwen3 | Reasoning Stage (S2) | midtrain | ~5T tokens at 4K with more STEM, code, reasoning and synthetic data; LR decay accelerated. Midtrain: a named stage that changes the mix and carries the decay, though the lab calls it pre-training. ↗ |
| Qwen | Qwen3 | Long Context Stage (S3) | midtrain | Hundreds of billions of tokens at 32,768 (75% 16K-32K docs); RoPE base 10k -> 1M via ABF; YaRN + DCA for 4x at inference. Midtrain: named context extension. ↗ |
| Qwen | Qwen3 | Stage 1: Long-CoT Cold Start | posttrain | Small long-CoT SFT on verified math/code/logic/STEM queries with QwQ-32B responses, deliberately few samples and steps. ↗ |
| Qwen | Qwen3 | Stage 2: Reasoning RL | posttrain | GRPO on 3,995 query-verifier pairs, large batches, many rollouts, off-policy updates, controlled entropy; AIME'24 70.1 -> 85.1 in 170 steps. ↗ |
| Qwen | Qwen3 | Stage 3: Thinking Mode Fusion | posttrain | SFT mixing self-generated thinking data with curated non-thinking data; /think and /no_think template; thinking budget emerges. ↗ |
| Qwen | Qwen3 | Stage 4: General RL | posttrain | RL over 20+ tasks (IF, format, preference, agent, RAG) with rule-based, reference-graded and RM rewards. ↗ |
| Qwen | Qwen3 | Strong-to-Weak Distillation | posttrain | For 0.6B-14B and 30B-A3B: off-policy distillation of think/no_think outputs, then on-policy logit KL to Qwen3-32B or 235B-A22B (~1/10 the GPU hours of RL). ↗ |
| Qwen | Qwen3-Coder | Pre-Training | pretrain | 7.5T tokens at 70% code, Qwen2.5-Coder-cleaned data, 256K native context with repo-scale and PR data; stage split and schedule not given. ↗ |
| Qwen | Qwen3-Coder | Scaling Code RL | posttrain | Execution-driven RL on diverse real-world coding tasks with automatically scaled test cases. ↗ |
| Qwen | Qwen3-Coder | Scaling Long-Horizon RL (Agent RL) | posttrain | Multi-turn tool-use RL on SWE-style tasks across 20,000 parallel environments on Alibaba Cloud. ↗ |
| Qwen | Qwen3-Next | Pre-training (15T) | pretrain | 15T tokens uniformly sampled from Qwen3's corpus, under 80% of Qwen3-30B-A3B's GPU hours; context schedule, optimizer and LR not given. ↗ |
| Qwen | Qwen3-Next | Post-training (Instruct and Thinking) | posttrain | Two separate post-trained models; RL with GSPO to handle hybrid-attention + sparse-MoE instability; no SFT or RL data details. ↗ |
| Qwen | Qwen3-Omni | AuT audio encoder training | midtrain | New ~0.6B audio encoder trained from scratch on 20M hours (ASR + audio understanding) to replace Whisper. Midtrain by rule (3): a new encoder for an existing LM. ↗ |
| Qwen | Qwen3-Omni | Encoder Alignment Stage (S1) | midtrain | LLM initialized from Qwen3 and frozen; vision (from Qwen3-VL) and AuT adapters, then encoders, trained. Midtrain by rule (3). ↗ |
| Qwen | Qwen3-Omni | General Stage (S2) | midtrain | All parameters, ~2T tokens: text 0.57T, audio 0.77T, image 0.82T, video 0.05T, video-audio 0.05T. Midtrain by rule (3). ↗ |
| Qwen | Qwen3-Omni | Long Context Stage (S3) | midtrain | 8,192 -> 32,768 with more long audio and long video. Midtrain: named context extension. ↗ |
| Qwen | Qwen3-Omni | Thinker: SFT, Strong-to-Weak Distillation, GSPO | posttrain | Lightweight SFT; off- then on-policy distillation from Qwen3-32B/235B-A22B; GSPO RL with rule-based and Qwen3 / Qwen2.5-VL judge rewards. ↗ |
| Qwen | Qwen3-Omni | Talker four-stage training | posttrain | Speech mapping on hundreds of millions of samples, CPT + long context, multilingual DPO, speaker fine-tuning. ↗ |
| Qwen | Qwen3-Omni | Captioner fine-tuning | posttrain | Audio-captioning variant fine-tuned from Qwen3-Omni-30B-A3B. ↗ |
| Qwen | Qwen3-VL | Vision encoder continued training (SigLIP-2) | midtrain | SigLIP-2 continued at dynamic resolution before joint training. Midtrain by rule (3): continued training of an earlier model. ↗ |
| Qwen | Qwen3-VL | Stage 0: Vision-Language Alignment | midtrain | 67B tokens at 8,192 (captions, visual knowledge, OCR), merger only. Midtrain by rule (3): adding vision to Qwen3. ↗ |
| Qwen | Qwen3-VL | Stage 1: Multimodal Pre-Training | midtrain | ~1T tokens at 8,192, all parameters, VL + text-only data, some video. Midtrain by rule (3). ↗ |
| Qwen | Qwen3-VL | Stage 2: Long-Context Pre-Training | midtrain | ~1T tokens at 32,768; more text-only, video and agent instruction data. Midtrain: context extension. ↗ |
| Qwen | Qwen3-VL | Stage 3: Ultra-Long-Context Adaptation | midtrain | 100B tokens at 262,144 emphasizing long video and long documents. Midtrain: context extension. ↗ |
| Qwen | Qwen3-VL | Supervised Fine-Tuning (incl. Long-CoT cold start) | posttrain | ~1.2M samples (1/3 text), 32K epoch then 256K epoch; separate non-thinking and long-CoT cold-start data for thinking models. ↗ |
| Qwen | Qwen3-VL | Strong-to-Weak Distillation | posttrain | Off-policy then on-policy logit-KL distillation on text-only data into the LLM backbone of smaller models. ↗ |
| Qwen | Qwen3-VL | Reasoning RL | posttrain | SAPO on ~30K verifiable text and multimodal queries (math, code, logic, grounding, puzzles), mixed-task batches, language-mismatch penalty. ↗ |
| Qwen | Qwen3-VL | General RL | posttrain | Multi-task RL on IF and preference with rule rewards and Qwen2.5-VL-72B / Qwen3 judges; targeted data to unlearn flawed priors and repetition. ↗ |
| Qwen | Qwen3-VL | Thinking with Images | posttrain | ~10k cold-start agent SFT + tool RL on Qwen2.5-VL-32B, distilled into ~120k interactions for Qwen3-VL SFT and tool-integrated RL. ↗ |
| Qwen | Qwen3-Coder-Next | Mid-training (continued pretraining on the Qwen3-Next base) | midtrain | Trillions of tokens of code, repos (~600B), PRs, rewritten web text, synthetic QA and agent trajectories; context 32K -> 262K; NTP + FIM; best-fit packing. Midtrain: named mid-training of an earlier base. ↗ |
| Qwen | Qwen3-Coder-Next | Supervised Fine-tuning | posttrain | SFT on in-house corpora, execution-verified agent trajectories and doc-grounded QA, filtered by a user-simulator agent and ranked by a pairwise judge. ↗ |
| Qwen | Qwen3-Coder-Next | Expert Models (WebDev, UX, single-turn RL, SWE RL) | posttrain | Four experts from the SFT model: WebDev (render-verified data), UX (21 tool templates), single-turn execution RL, multi-turn SWE RL with turn and tool-format penalties. ↗ |
| Qwen | Qwen3-Coder-Next | Expert Distillation | posttrain | The four experts are distilled back into the SFT model to give one deployable model without expert routing. ↗ |
| Qwen | WebWorld | Stage 1: Transition Modeling | posttrain | Full-parameter SFT of Qwen3 8B/14B/32B on 1.06M (state, action -> next state) trajectories, 1 epoch, LR 2e-5, 20K-token packing, 16xA100. Posttrain: the lab calls it SFT. ↗ |
| Qwen | WebWorld | Stage 2: Reasoning Activation | posttrain | 1K synthesized CoT samples from the Stage 1 checkpoint, 5 epochs, LR 8e-6, CoT-only loss, 8xA100 for 2-3 hours. Posttrain: SFT. ↗ |
| Qwen | Qwen3.5 | Pretraining (native multimodal, early fusion) | pretrain | From-scratch early-fusion text+vision pretraining on trillions of tokens, 201 languages, FP8 pipeline; no separate mid-training phase described. ↗ |
| Qwen | Qwen3.5 | Post-training: scaled asynchronous RL | posttrain | Disaggregated async RL over million-scale agent scaffolds and environments for all sizes, text and multimodal; FP8 end-to-end, rollout router replay. SFT details not given. ↗ |
| Qwen | Qwen-AgentWorld | Stage 1: Continual Pre-Training (CPT) | midtrain | Continued pre-training of Qwen3.5 base checkpoints on non-thinking environment trajectories (10M+ over 7 domains) plus domain-knowledge corpora, with turn-level loss masking. Midtrain: named continued pre-training. ↗ |
| Qwen | Qwen-AgentWorld | Stage 2: Supervised Fine-Tuning | posttrain | 7,094 rejection-sampled thinking trajectories (best of 3 rollouts, judge-scored), 256k context, prompts from 10 template variants. Posttrain: SFT. ↗ |
| Qwen | Qwen-AgentWorld | Stage 3: Reinforcement Learning | posttrain | GSPO on 92,308 single-turn prediction samples (prompts up to 128k), 9:1 rubric-judge + rule-verifier reward; 440 steps tracked on 35B-A3B. Posttrain: RL. ↗ |
| Qwen | Qwen3.8 | Scaling Real-World RL Systems | posttrain | RL with environments scaled by task, workspace and harness, a universal reward system (execution + rubric + agentic checks) and an online batch balancer. Pretraining not described. ↗ |
| Qwen | Qwen3.8-Flash-Next | Pre-training (new architecture, Muon) | pretrain | From scratch with GDN + full attention 3:1, Gated Residual, n-gram embedding, Muon; target batch from step 0; ~1/3 the tokens and ~1/9 the FLOPs of Qwen3.7-Plus-Base. ↗ |
| Qwen | Qwen3.8-Flash-Next | CPT Stage 1: Dense Distillation (QSA indexer warm-up) | midtrain | At 256K, only the QSA indexer learns to match full-attention scores (KL): 1,000 steps, LR 1e-3, ~2B tokens. Midtrain: named continued pre-training that retrofits attention. ↗ |
| Qwen | Qwen3.8-Flash-Next | CPT Stage 2: Sparse Training (final stage of CPT) | midtrain | Backbone and indexer trained jointly with QSA active: 8,000 steps, LR 2.5e-5, 96 × 256K tokens per step (~200B). Midtrain: named continued pre-training stage. ↗ |
| MiniMax | Text-01 | Initial Pre-training | pretrain | 8K sequences, AdamW, batch 16M -> 128M tokens; LR 2e-4 constant for 7.2T, then 1.3e-4 for 3.2T more after gradient-norm anomalies. Pretrain: the main constant-LR run. ↗ |
| MiniMax | Text-01 | fast decay phase | midtrain | 1T tokens, LR decayed exponentially to 3e-5. Midtrain: a named learning-rate decay phase. ↗ |
| MiniMax | Text-01 | Long-Context Extension (three stages) | midtrain | 128K (RoPE 5M, 300B tokens) -> 512K (10M, 32B) -> 1M (10M, 26B); 10% long-context QA in the last 20% of each stage. Midtrain: a named context-extension phase. ↗ |
| MiniMax | Text-01 | Stage I-II: short- then long-context SFT | posttrain | SFT at 8K, then SFT at 1,032,192 tokens with 50% long-context prompts; RoPE base 10M throughout post-training. ↗ |
| MiniMax | Text-01 | Stage III-IV: short- then long-context DPO | posttrain | Offline DPO at 8K, then DPO on entirely long-context data at 1,032,192 tokens; pairs are best vs worst reward-model-scored samples. ↗ |
| MiniMax | Text-01 | Stage V: Online Reinforcement Learning | posttrain | Online RL at 8K with modified GRPO, mainly math, on SFT-untrained prompts with moderate success rates. ↗ |
| MiniMax | VL-01 | Vision encoder pre-training | midtrain | 303M ViT-L/14 trained from scratch with CoCa-style contrastive + captioning losses on 694M unique pairs. Midtrain: part of adding vision to the finished Text-01 LM. ↗ |
| MiniMax | VL-01 | Stage I: Modality alignment | midtrain | ViT and MLP adapter trained on 80B description tokens at fixed 336x336 to caption images. Midtrain: continued training that adds vision to a finished LM. ↗ |
| MiniMax | VL-01 | Stage II: Enhancement of Vision Understanding | posttrain | 420B multimodal instruction tokens mixed 20:1 with Text-01 post-training data, all parameters. Posttrain: the lab calls it instruction tuning, though it carries most of the vision tokens. ↗ |
| MiniMax | VL-01 | Stage III: Enhancement of User Experience | posttrain | 44.8B tokens of labeled multi-turn conversations on app-sourced images, one epoch. Posttrain: SFT. ↗ |
| MiniMax | VL-01 | Stage IV: Enhancement of Preference | posttrain | DPO on 40,000 image-text pairs plus text pairs, judged by Text-01, stopped before one full epoch. Posttrain: preference optimization. ↗ |
| MiniMax | SynLogic | Zero-RL on SynLogic data | posttrain | GRPO directly on Qwen2.5 base models with binary format/correctness rewards: 33k hard logic samples (32B); Mix-3 adds 35k math and 9k code. Posttrain: RL on a borrowed base. ↗ |
| MiniMax | M1 | Continual Pre-Training | midtrain | 7.5T tokens on Text-01: LR 8e-5 constant for 2.5T, then decay to 8e-6 over 5T; STEM/code/book/reasoning share raised to 70%, no synthetic data. Midtrain: continued pre-training of an earlier base. ↗ |
| MiniMax | M1 | Long Context Extension | midtrain | Four stages from 32K to 1M, made smoother because aggressive extension caused gradient explosions. Midtrain: a named context-extension phase. ↗ |
| MiniMax | M1 | Supervised Fine-Tuning | posttrain | Cold-start long-CoT SFT (reflection-style reasoning) across math, code, STEM, writing, QA and chat; math and code about 60%. ↗ |
| MiniMax | M1 | RL with CISPO (40K) | posttrain | CISPO over math (~50K), logic (~53K), code (30K), SWE sandboxes and 25K general tasks with a rule-to-general curriculum; yields M1-40k. ↗ |
| MiniMax | M1 | Extending RL Scaling to Longer Thinking (80K) | posttrain | Harder data filtered with the 40K model, synthetic reasoning data down-sampled, output window grown 40K -> 80K in 8K steps; yields M1-80k. Whole RL: 3 weeks on 512 H800s. ↗ |
| MiniMax | M2 | constant phase of pre-training | pretrain | 19.9T tokens at 8K with code, math and STEM upsampled and a single MTP module (loss weight 0.3). Pretrain: the main constant-LR run. ↗ |
| MiniMax | M2 | decay phase (long-context extension) | midtrain | 9.3T tokens of short-text decay data + long-context data, context 8K -> 32K -> 192K, MTP grown to 3 modules by weight copying. Midtrain: a named decay phase that also extends context. ↗ |
| MiniMax | M2 | Supervised Fine-Tuning | posttrain | Rejection-sampled interleaved-thinking trajectories over chat, reasoning, code and cowork, with multi-stage cleaning. ↗ |
| MiniMax | M2 | Reinforcement Learning (Forge, CISPO) | posttrain | Agent RL with CISPO in the Forge system: process, completion-time and outcome rewards; per-stage shifts toward agent/coding data, longer contexts and harder tasks. ↗ |
| MiniMax | M2.1 | post-training update of M2 | posttrain | Further post-training of M2 for multi-language coding, Android/iOS/web, office tasks and more concise thinking; no new pre-training described. ↗ |
| MiniMax | M2.5 | RL scaling in Forge | posttrain | Large-scale agentic RL (CISPO, process + completion-time rewards) over 200K+ coding environments and expert-built office tasks. ↗ |
| MiniMax | M2.7 | self-evolving post-training | posttrain | Continued M2-series post-training in which an internal M2.7 ran 30-50% of the RL team's experiment loop; adds full-stack repo, office, Agent Teams and persona data. ↗ |
| MiniMax | M3 | native multimodal pre-training | pretrain | Text, image and video mixed from the first step with MSA attention and a rebuilt interleaved-data pipeline; token count and schedule not disclosed. Pretrain. ↗ |
| MiniMax | M3 | agentic post-training with a user simulator | posttrain | Coding-agent training against simulated developers (clarification, feedback, task switching); MiniMax Code harness trained together with M3; algorithm not disclosed. ↗ |
| MiniMax | M3.1-Flash | sparse attention throughout | midtrain | M3's three full-attention layers converted to MSA; how the conversion was trained is not described. Midtrain: retrofitting attention on an earlier base (inferred). ↗ |
| MiniMax | M3.1-Flash | Quantization-aware training | posttrain | QAT of routed experts (NVFP4 in the release) and attention paths (MXFP8); training stage not specified. Posttrain: most QAT. ↗ |
| StepFun | ORZ | Reasoner-Zero RL from the base model | posttrain | PPO with GAE(1,1), no KL, binary correctness reward, 128 prompts x 64 responses per step on Qwen2.5 bases (0.5B-32B). Posttrain: RL on a finished base. ↗ |
| StepFun | ORZ | ORZ-32B 'annealing' stage | posttrain | 100 extra RL steps on 13k hard prompts (<4/64 correct in the first 1,100 steps), LR decayed linearly to 3e-7. Posttrain: an RL phase the lab calls annealing. ↗ |
| StepFun | Step-3 | Pretraining | pretrain | 20T+ text tokens in 10+ languages plus 4T image-text mixed tokens; schedule, optimizer and context length not disclosed. ↗ |
| StepFun | Step-3 | Annealing phases | midtrain | 3.7T high-quality text tokens from a domain-balanced, down-sampled annealing set. Midtrain: a phase the lab names annealing. ↗ |
| StepFun | Step-3 | Supervised fine-tuning | posttrain | SFT on filtered multimodal math, competitive programming, STEM and general conversations, plus reverse-synthesized reasoning and tool-use queries. ↗ |
| StepFun | Step-3 | Reinforcement learning | posttrain | RL with a value network: verified rewards on difficulty-balanced math, code, logic and multimodal perception prompts; preference model or human feedback for open-ended tasks. ↗ |
| StepFun | PaCoRe-8B | RLVR-8B start checkpoint | posttrain | Reasoning-oriented post-training of Qwen3-8B-Base (details not given), released as RLVR-8B-0926. ↗ |
| StepFun | PaCoRe-8B | PaCoRe RL Stage 1 | posttrain | 250 strict on-policy PPO iterations synthesizing 16-24 cached messages; 16 instances x 64 responses, 131K max length. ↗ |
| StepFun | PaCoRe-8B | PaCoRe RL Stage 2 | posttrain | 450 more PPO iterations on a further-refined data distribution. ↗ |
| StepFun | Step-3.5-Flash | Pre-training Stage 1: Open-domain pre-training | pretrain | 14.6T tokens at 4K on the broad mix; Muon, 2k-step warmup to 2.5e-4 then cosine to 5e-5; batch ramps 4,096 -> 16,384 sequences over the first 400B tokens. ↗ |
| StepFun | Step-3.5-Flash | Pre-training Stage 2: Annealing + long-context initialization | midtrain | 3T tokens: 2T at 4K (LR 5e-5 -> 2e-5) then 1T at 32K (LR 2e-5, full-attn RoPE 1M) on a mix annealed to code/PRs and quality knowledge. Midtrain: a phase the lab names annealing. ↗ |
| StepFun | Step-3.5-Flash | Mid-training Stage 1: Specialization at 32k | midtrain | 386B tokens at 32K including 81B (21%) replay; SWE and tool-use-centric mix; router frozen, LR warm to 2e-5 then constant. Midtrain: named mid-training. ↗ |
| StepFun | Step-3.5-Flash | Mid-training Stage 2: Long-context specialization | midtrain | 364B tokens at 128K (full-attn RoPE 5M) with 10.5B replay; synthetic long-horizon reasoning, >32K documents, agent data; LR to 7.3e-6. Midtrain: named mid-training and context extension. ↗ |
| StepFun | Step-3.5-Flash | Two-stage SFT | posttrain | Stage 1: 871k samples / 7.23B tokens multi-domain at 128K; Stage 2: ~30k expert chemistry trajectories and synthetic arithmetic for three epochs. ↗ |
| StepFun | Step-3.5-Flash | Expert models via domain-specific RL | posttrain | Separate RL runs from the shared SFT model for math, code, STEM, tool use, long context, human preference and agentic reasoning. ↗ |
| StepFun | Step-3.5-Flash | Self-distillation into a unified student | posttrain | Experts generate trajectories on the SFT prompt distribution; rejection sampling removes language mixing and overthinking; one student trained from the mid-train checkpoint. ↗ |
| StepFun | Step-3.5-Flash | Scalable RL (MIS-PO) | posttrain | RLVR + GenRM RLHF + agent rewards with MIS-PO, truncation-aware value bootstrapping, 128K rollouts and unbiased KL 0.001 in the final stage. ↗ |
| StepFun | Step-3.5-Flash | MTP-2/3 joint training | posttrain | MTP-2 and MTP-3 cloned from MTP-1 and trained jointly, with position-dependent loss weights, in a short final phase. ↗ |
| StepFun | Step-5-Preview | Pre-training (FP8, deep stack) | pretrain | From-scratch FP8 training of the 92-block 600B MoE with FP32 hidden states, FP32 RMSNorm scales and fine-grained gradient clamping; tokens and schedule not disclosed. ↗ |
| StepFun | Step-5-Preview | SFT, then MTP-3 head | posttrain | SFT (not described), then an MTP-3 head trained on top, giving roughly 30% end-to-end speedup in RL rollouts. ↗ |
| StepFun | Step-5-Preview | Agentic RL | posttrain | Long-horizon RL on million-level verifiable environments with bitwise train-inference alignment, routed-expert replay, pretrained critic, turn-level penalties and learned compaction. ↗ |
| Ling | Ling-plus | Initial Pre-Training | pretrain | 9T tokens at 4K, AdamW, WSD with peak 2.4e-4 halved at ~60% of tokens, batch 2,560 -> 8,960, mix re-weighted toward high-quality data during the run. ↗ |
| Ling | Ling-plus | Long Context Pre-Training | midtrain | 150B tokens at 16K with RoPE θ 10K -> 600K; less web, more long-form (Plus) or math/code (Lite); LR schedule unchanged. Midtrain: a named context-extension phase. ↗ |
| Ling | Ling-plus | Annealing | midtrain | Inverse-square-root decay 1.2e-4 -> 1.2e-8 on curated high-quality data; token count not stated. Midtrain: a named annealing phase. ↗ |
| Ling | Ling-plus | Supervised Fine-Tuning + Direct Preference Optimization | posttrain | Iterative SFT (1-2M seed scaled by synthesis; short then 4K-16K data at 95:5) followed by vanilla, robustness and format-only DPO rounds; tool-use data from knowledge graphs. ↗ |
| Ling | Ling-Coder-lite | Continuous Pre-Training (Stage 1, Stage 2) | midtrain | From a 7T Ling-lite checkpoint: 870B then 1.7T tokens, code ≥60%, NTP + FIM, 16M-token batches. Midtrain: continued training of an earlier base. ↗ |
| Ling | Ling-Coder-lite | Annealing | midtrain | 630B tokens, inverse-sqrt LR 1.4e-4 -> 1.4e-6, less raw code and much more synthetic QA. Midtrain: a named annealing phase. ↗ |
| Ling | Ling-Coder-lite | Supervised Fine-tuning; Direct Preference Optimization | posttrain | SFT on 18.8M instructions for 5 epochs (LR 4e-5), then offline DPO on 1.4M pairs for 2 epochs (LR 3e-7). ↗ |
| Ling | Ring-lite-distill-preview | Reasoning-oriented SFT; Generalization-oriented SFT; DPO | posttrain | SFT on ~2.59M R1-distilled reasoning samples, then a balanced SFT with 1,082K general samples, then 7K-sample DPO for format and instruction adherence. ↗ |
| Ling | Ring-lite | Long-CoT SFT (Ring-lite-distill) | posttrain | Long-CoT SFT of Ling-lite-1.5: 3 epochs, LR 3e-4 cosine, batch 256, 32K context with RoPE base 600K; checkpoint chosen by entropy. ↗ |
| Ling | Ring-lite | Two-stage RL Training; General SFT | posttrain | C3PO RL on math, then code + science (max length 24,576 -> 32,768), then a general SFT stage for instruction following, writing and safety. ↗ |
| Ling | Ring-lite-2507 | Reasoning RL; General RL | posttrain | Reasoning RL on 67K math, 32K code, 9.9K science and 19K+ logic problems, then general RL with a reward model and rule verifiers. ↗ |
| Ling | Ring-mini-2.0 | Long-CoT SFT + RLVR + RLHF | posttrain | Long-CoT SFT on Ling-mini-2.0-base, then RLVR and RLHF optimised jointly. ↗ |
| Ling | Ring-flash-2.0 | SFT + RLVR + RLHF Multi-Stage Training | posttrain | Light Long-CoT SFT, RLVR with IcePop, then a separate RLHF stage (two-staged RL chosen over joint training for efficiency). ↗ |
| Ling | Ring-linear-2.0 | Continued training stage | midtrain | Ling 2.0 bases converted to hybrid linear attention, then 600B (mini) / 1T (flash) tokens at 4K. Midtrain: retrofitting attention onto an earlier base. ↗ |
| Ling | Ring-linear-2.0 | Mid-training stage | midtrain | Context 4K -> 32K -> 128K with more reasoning data, WSM checkpoint merging; tokens not stated. Midtrain: named mid-training with context extension. ↗ |
| Ling | Ring-linear-2.0 | Supervised Fine-Tuning; Reinforcement Learning | posttrain | 128K SFT on reasoning + general data, then RL at 64K across math, code, science, logic and subjective tasks after module-by-module train/inference alignment. ↗ |
| Ling | Ling-1T | Pre-Training Substage 1 and Substage 2 (4K) | pretrain | 20T tokens at 4K in two 10T sub-stages, reasoning share 32% -> 46%, stricter decontamination; AdamW, constant LR after warm-up (WSM), FP8. ↗ |
| Ling | Ling-1T | Mid-Training: Long Context Extension (32K) | midtrain | 150B tokens with 20% 32K sequences, 4K -> 32K; YaRN later extends to 128K. Midtrain: named mid-training and context extension. ↗ |
| Ling | Ling-1T | Mid-Training: Reasoning Pre-Activation + WSM merge | midtrain | 600B tokens with high reasoning share plus CoT corpora; top-32 checkpoints averaged in place of LR decay. Midtrain: named mid-training with data change. ↗ |
| Ling | Ling-1T | Decoupled Fine-Tuning; Evolutional RL (Evo-CoT); RL with Group Arena Reward | posttrain | DFT with instant vs in-depth system prompts; Evo-CoT RL with LPO and length/format rewards in instant mode; RLHF with Group Arena Reward and RubriX. ↗ |
| Ling | Ring-1T | Long-CoT SFT | posttrain | Long-CoT SFT of Ling-1T-base on math 46%, STEM 26%, code 20%, other 8%; 64K packing, 3 epochs. ↗ |
| Ling | Ring-1T | Reasoning RL; General RL | posttrain | Reasoning RLVR with IcePop + C3PO++ (480 prompts x 8, 64K), then general RLHF with GRPO (80 x 8, 32K). ↗ |
| Ling | Ring-mini-sparse-2.0-exp | Continued training with MoBA | midtrain | Ling-mini-base-2.0-20T continued 100B tokens with MoBA block-sparse attention; YaRN 4x to 128K. Midtrain: retrofitting attention onto an earlier base. ↗ |
| Ling | Ling-2.5-1T | Incremental training to hybrid linear attention + continued pre-training | midtrain | Ling-2.0-1T GQA converted to 1:7 MLA + Lightning, then 9T tokens for knowledge and agent ability with training context extended to 256K. Midtrain: retrofit of an earlier base. ↗ |
| Ling | Ling-2.6-1T | Migration Pre-Training | midtrain | ~400B tokens at 4K: Lightning conversion, linear warm-up (only Wqkv/QK-Norm trainable), QK-Norm absorption + TransMLA, MLA warm-up. Midtrain: retrofitting attention. ↗ |
| Ling | Ling-2.6-1T | Continue Pre-Training | midtrain | 8T tokens at 4K, ~46% reasoning / 50% general / 4% multilingual, new high-quality data from the outset. Midtrain: continued pre-training of an earlier base. ↗ |
| Ling | Ling-2.6-1T | Mid-Training | midtrain | ~1.2T tokens: 250B (20% at 32K, peak LR), 425B at 32K with more high-quality data, 525B at 256K with 15% agentic data; WSM merge. Midtrain: named mid-training. ↗ |
| Ling | Ling-2.6-1T | Cold-Start SFT; Specialist training; Specialist distillation; Bidirectional preference alignment | posttrain | Cold-start SFT, expert SFT + RL per specialist (Evo-CoT/LPO for reasoning, GSPO for agents), distillation into one model, bidirectional focus-reward alignment; MTP continued training. ↗ |
| Ling | Ring-2.6-1T | Cold-Start SFT; Reasoning & Agentic RL (KPop); Specialist distillation; Adaptive thinking | posttrain | Cold-start SFT, reasoning and agentic specialists trained with async KPop RL, specialist distillation, then adaptive-thinking SFT + RL for high/xhigh effort. ↗ |
| Ling | Ling-3.0-flash | Pre-trained checkpoint (Ling-3.0-flash-base-30T) | pretrain | From-scratch pre-training of the native KDA + gated MLA model, ~30T tokens by checkpoint name, 8K context, no LR decay (WSM). ↗ |
| Ling | Ling-3.0-flash | Mid-trained checkpoint; Merged checkpoint (WSM) | midtrain | Named mid-training (context 8K -> 32K -> 256K; data not described) then weighted checkpoint merging in place of LR decay. Midtrain: named mid-training phase. ↗ |
| Ling | Ling-3.0-flash | Agentic post-training | posttrain | Post-training with 10,000+ interactive environments for coding, general and deep-research agents; hybrid thinking on by default. Stages and algorithm not described. ↗ |
| Ling | Ling-3.0-flash-Fin | Continued training on financial data | posttrain | Domain continued training starting from the post-trained Ling-3.0-flash (not its base); data and method beyond 'high-quality financial data' not described. ↗ |
| Ling | Ling-3.0-flash-VL | Vision extension of Ling-3.0-flash | midtrain | ViT encoder + MLP projector added to Ling-3.0-flash for image and video input at 256K; training stages not described. Midtrain: adding vision to a finished LM. ↗ |
| Nemotron | Llama-Nemotron | Stage 1: NAS (Puzzle) and FFN Fusion | midtrain | Puzzle block-wise local distillation from Llama 3.3-70B / 3.1-405B Instruct, MIP block selection; FFN Fusion for Ultra. Midtrain: retrofits the architecture of a finished Llama model. ↗ |
| Nemotron | Llama-Nemotron | Stage 2: recovery training (knowledge distillation and continued pretraining) | midtrain | Super 40B tokens KD; Ultra 65B tokens KD then 88B tokens CPT on Nemotron-H phase 4 data. Midtrain: named continued pre-training of an earlier base. ↗ |
| Nemotron | Llama-Nemotron | Stage 3: supervised fine-tuning | posttrain | 33M-sample synthetic set with 'detailed thinking on/off'; Nano 3-stage SFT (32k packing), Super 1 epoch at 16k, Ultra 24k packing with LR 1e-5 cosine to 1e-6. Posttrain: SFT. ↗ |
| Nemotron | Llama-Nemotron | Stage 4: large-scale RL for reasoning | posttrain | GRPO on LN-Ultra only (~140k H100 hours, FP8 rollouts, pass-rate curriculum); smaller models rely on distillation. Posttrain: RL. ↗ |
| Nemotron | Llama-Nemotron | Stage 5: alignment (instruction following and human preference) | posttrain | RLOO instruction-following RL (Super, Ultra); RLHF with online RPO (Super), GRPO (Ultra), offline RPO (Nano); Super checkpoints merged. Posttrain: RL. ↗ |
| Nemotron | Nemotron-H | Phase 1 (0-60% of training) | pretrain | Diversity blend (web crawl 59%, code 20%, academic 8.8%, multilingual 5%) at 8192 tokens, batch 768, warmup then cosine LR (4e-4 for 56B); 56B in FP8 ↗ |
| Nemotron | Nemotron-H | Phases 2-3 (60%-80%, 80% to last 380B tokens) | pretrain | Shift to high-quality data (math 14%, academic 12%, wiki up); phase 3 adds 12.6% SFT-style data. Pretrain: data re-weighting inside the main cosine schedule. ↗ |
| Nemotron | Nemotron-H | Phase 4 (last 380B tokens) | pretrain | SFT-style data raised to 31.8%, web crawl cut to 13.5%. Pretrain: final data re-weighting within the single cosine run, not named as a separate stage. ↗ |
| Nemotron | Nemotron-H | MiniPuzzle compression (56B -> 47B-Base) | midtrain | NAS-selected layer + FFN pruning, ~7B-token KD per candidate, then 63B-token forward-KL distillation in FP8 at 8192, batch 768. Midtrain: continued training of a finished base into a new base. ↗ |
| Nemotron | OpenCodeReasoning-Nemotron | SFT on OpenCodeReasoning | posttrain | SFT of Qwen2.5 7B/14B/32B (base and instruct) on 736K DeepSeek-R1 Python solutions; 3 epochs, batch 256, 32,768 max length, LR 5e-5. No RL. Posttrain: SFT. ↗ |
| Nemotron | OpenMath-Nemotron | First SFT round (CoT + TIR + GenSelect) | posttrain | 5.5M samples (3.2M CoT, 1.7M TIR, 566K GenSelect), 6 epochs, batch 1024, LR 3e-4 (1.5B) to 1e-4 (14B/32B), 4-checkpoint averaging. Posttrain: SFT. ↗ |
| Nemotron | OpenMath-Nemotron | Second SFT round on harder problems | posttrain | 2.2M samples on olympiad-forum problems hard for Qwen2.5-Math-72B-Instruct TIR, solutions >=5000 tokens; 4 epochs; skipped for 32B. Posttrain: SFT. ↗ |
| Nemotron | LN-Nano-4B-v1.1 | SFT stages | posttrain | Multi-stage SFT of the Minitron 4B base on reasoning on/off data (math, code, science, safety, tool calling), sequence-packed. Posttrain: SFT. ↗ |
| Nemotron | LN-Nano-4B-v1.1 | multiple RPO stages | posttrain | Several RPO rounds on on-policy data; final checkpoint merges SFT and RPO checkpoints. Posttrain: preference optimization. ↗ |
| Nemotron | LN-Nano-4B-v1.1 | RoPE scaling + reasoning accuracy recovery RPO round | posttrain | RoPE scaling for 128K, then a light RPO round to recover reasoning accuracy. Posttrain: comes after RPO and the only training described is an RPO round. ↗ |
| Nemotron | AceReason-Nemotron | AceReason-Nemotron-1.1: SFT from Qwen2.5-Math-7B | posttrain | 1.1 only: SFT on DeepSeek-R1 responses (383K math+code prompts, blends up to 2.2M samples), rope_theta raised to 1M for 128K. Posttrain: SFT. ↗ |
| Nemotron | AceReason-Nemotron | Math-only RL (8K→16K→24K) | posttrain | GRPO strict on-policy, no KL; batch 128, 8-16 rollouts, LR 1e-6; harder prompts (pass rate <=6/16) at 24K. Starts from R1-Distill-Qwen (1.0) or own SFT (1.1). Posttrain: RL. ↗ |
| Nemotron | AceReason-Nemotron | Code-only RL (24K→32K) | posttrain | Two-stage code RL on 8,520 curated competitive-programming problems, 24K then 32,768 max response, epoch-wise filtering of easy problems. Posttrain: RL. ↗ |
| Nemotron | AceReason-Nemotron | Final math-only RL (32K) | posttrain | Closing math-only RL round at 32K response length. Posttrain: RL. ↗ |
| Nemotron | LN-Nano-VL | interleaved image-text pre-training (three training stages) | midtrain | Vision training of Llama-3.1-8B-Instruct with C-RADIOv2-H on commercial images, interleaved image-text with LLM unfrozen; stages not itemised. Midtrain: adds vision to a finished LM. ↗ |
| Nemotron | LN-Nano-VL | SFT (NV-CosmosNemotron-SFT) | posttrain | Instruction tuning on NV-CosmosNemotron-SFT with text-only instruction data re-blended in. Posttrain: SFT. ↗ |
| Nemotron | Nemotron-H-Reasoning | SFT Stage 1: math, code and science reasoning | posttrain | 5:1 reasoning:non-reasoning samples with <think> traces plus paired no-reasoning answers; blog: ~30,000 steps at batch 256 ↗ |
| Nemotron | Nemotron-H-Reasoning | SFT Stage 2: instruction following, dialogue and safety | posttrain | ~10x smaller blend, balanced reasoning/non-reasoning, improves reasoning-mode control ↗ |
| Nemotron | Nemotron-H-Reasoning | Long-context training | posttrain | A few hundred steps on concatenated chats with synthetic long-range references, doc QA and keyword retrieval; 83% RULER@128k. Posttrain: an SFT-format phase after SFT. ↗ |
| Nemotron | Nemotron-H-Reasoning | Reinforcement learning with GRPO | posttrain | IFEval RL (16K LMSYS prompts, rule verifier, LR 1e-6), tool calling with abstention (~50K samples, blog), then ~200 GRPO steps on HelpSteer2 with a Qwen-32B reward model ↗ |
| Nemotron | Nemotron-H-Reasoning | Checkpoint merging (8B) | posttrain | 8B only: blend with an earlier, safer RL checkpoint at alpha ~0.5 to fix weak internal safety scores ↗ |
| Nemotron | LN-Super-v1.5 | NAS + knowledge distillation (KD) | midtrain | Same Puzzle-NAS 49B architecture from Llama-3.3-70B-Instruct with 40B-token KD (FineWeb, Buzz-V1.2, Dolma), as in v1. Midtrain: continued training of an earlier base. ↗ |
| Nemotron | LN-Super-v1.5 | supervised fine-tuning stage | posttrain | SFT on synthetic data from Qwen3 235B and DeepSeek R1 0528 (Nemotron-Post-Training-Dataset-v1, 26M+ rows). Posttrain: SFT. ↗ |
| Nemotron | LN-Super-v1.5 | RL stages: RPO, RLVR, iterative DPO | posttrain | RPO for chat, RLVR for reasoning, iterative DPO for tool calling; final model merges several RL and DPO checkpoints. Posttrain: RL. ↗ |
| Nemotron | Nano-2 | Pretraining Phase 1 (0-60%) | pretrain | Diverse blend (crawl tiers ~60%, code 20%, multilingual 5%) at 8192, batch 736, WSD warmup (8B tokens) then stable LR 4.5e-4, FP8 throughout ↗ |
| Nemotron | Nano-2 | Pretraining Phase 2 (60-90%) | pretrain | Higher-quality blend: syn-crawl-high 21%, code 20%, crawl-high 16%, stem-sft 14.5%, math 9.5%. Pretrain: re-weighting inside the main WSD run. ↗ |
| Nemotron | Nano-2 | Pretraining Phase 3 (90-100%) and WSD decay | pretrain | LR decays 4.5e-4 -> 4.5e-6 over the last 3.6T tokens (from 16.4T, computed); phase 3 from 18T: stem-sft 32%, code 16%, math 11%, code-sft 10.9%. Pretrain: decay inside the main schedule. ↗ |
| Nemotron | Nano-2 | Phase LC (long-context continuous pretraining) | midtrain | 18.9B tokens at 512k, batch 12 (~6M tokens), TP8 + CP16; Phase 3 blend at 80% + 20% synthetic long-doc QA. Midtrain: a named continued-pretraining context-extension phase. ↗ |
| Nemotron | Nano-2 | SFT Stage 1 | posttrain | ~80B tokens of SFT data packed into ~128k-token sequences; ~10% reasoning-off examples ↗ |
| Nemotron | Nano-2 | SFT Stage 2 (tool calling) | posttrain | Fixes tool-calling regression from 128k packing in Stage 1 ↗ |
| Nemotron | Nano-2 | SFT Stage 3 (long context + truncated traces) | posttrain | Nemotron-H long-context SFT recipe plus truncated-trace examples for thinking-budget robustness ↗ |
| Nemotron | Nano-2 | IFEval RL | posttrain | GRPO with a rule-based instruction verifier; careful checkpoint selection ↗ |
| Nemotron | Nano-2 | DPO (tool calling) | posttrain | Iterative on-policy DPO in WorkBench: successful vs failed multi-step tool-call trajectories ↗ |
| Nemotron | Nano-2 | RLHF | posttrain | GRPO with a Qwen-based reward model, rollouts with and without thinking, targeting Arena-Hard ↗ |
| Nemotron | Nano-2 | Model merging (12B) | posttrain | Interpolate reasoning-strong and chat-strong RL checkpoints at alpha ~0.5 to get the final 'Merged' 12B ↗ |
| Nemotron | Nano-2 | Pruning and distillation to Nemotron-Nano-9B-v2 (reasoning model) | posttrain | Depth 62->56 + KD ~60B@8K; width pruning + KD ~50B@8K, ~25B@49K, ~1B@262K; then DPO, GRPO, KD ~0.4B@262K, RLHF, 0.5 merge. Posttrain: distillation of the aligned model. ↗ |
| Nemotron | Nano-2 | Pruning and distillation to Nemotron-Nano-9B-v2-Base | midtrain | Then ~2.5B KD tokens at 524,288, all on pretraining data. Midtrain: continued training of a finished base into a smaller base. ↗ |
| Nemotron | Nano-2-VL | Stage 0 (connector warm-up; 'Pretraining' in Fig. 2) | midtrain | ~2.2M samples (up to 36B tokens) at 16K, only the MLP projector trained. Midtrain: adding vision to a finished LM. ↗ |
| Nemotron | Nano-2-VL | SFT Stage 1 (16K) | posttrain | All weights unfrozen; 26M multimodal samples (~72B tokens) + 6.5M Nano 2 text-reasoning samples (~40B); FP8 ↗ |
| Nemotron | Nano-2-VL | SFT Stage 2 (49K context extension) | posttrain | Video and multi-image data (1.4M samples, 17B tokens) + 25% Stage-1 reuse. Posttrain: an SFT stage after SFT Stage 1, although the lab calls it context extension. ↗ |
| Nemotron | Nano-2-VL | SFT Stage 3 (49K text recovery) | posttrain | Code-reasoning-only SFT to recover LiveCodeBench lost in Stages 1-2 ↗ |
| Nemotron | Nano-2-VL | SFT Stage 4 (300K context extension) | posttrain | Nano 2 Stage-3 long-context SFT data, 8-way context parallelism; lifts RULER. Posttrain: SFT stage after SFT Stage 1. ↗ |
| Nemotron | Nano-2-VL | Quantization-aware distillation (NVFP4) | posttrain | FP8 checkpoint via PTQ; NVFP4 student distilled from BF16 teacher with Stage-1 hyperparameters and LR 2e-6 ↗ |
| Nemotron | Nemotron-3-Nano | Pre-training Phase 1 (diverse data, 0-94%) | pretrain | 23.5T tokens of a diversity-oriented 15-category mix at 8K context on the WSD schedule (8.4B-token warmup, constant 1e-3 to 80%, then decay). Pretrain: the main LR schedule. ↗ |
| Nemotron | Nemotron-3-Nano | Pre-training Phase 2 (high-quality data, 94-100%) | pretrain | 1.5T tokens weighted to high-quality data (stem-sft 22.3%, math 12.5%, wiki) during the WSD decay to 1e-5. Pretrain: data re-weighting inside the main LR schedule. ↗ |
| Nemotron | Nemotron-3-Nano | Long-Context Extension (LC-Phase, continuous pretraining) | midtrain | 121B tokens at constant LR 1e-5, batch 48, mixing 512k and 4k sequences; 20% doc QA, 1% synthetic retrieval, 79% Phase-2 data. Midtrain: a named context-extension phase. ↗ |
| Nemotron | Nemotron-3-Nano | Supervised Fine Tuning | posttrain | 18M+ chat, code, math, science, agentic and multilingual samples; 13,000 steps, batch 64, packed to 256K, LR 5e-5; adds reasoning on/off and budget control. Posttrain: SFT. ↗ |
| Nemotron | Nemotron-3-Nano | Multi-environment RLVR (first stage) | posttrain | Synchronous GRPO over all NeMo Gym environments at once (math, code, STEM QA, IF, JSON, long context, tool use), 128 prompts × 16 rollouts, curriculum sampling. Posttrain: RL. ↗ |
| Nemotron | Nemotron-3-Nano | Reinforcement Learning from Human Feedback (GenRM) | posttrain | An RL-trained Qwen3-235B GenRM compares 16 responses per prompt in a circle; Group Relative Length Control cuts verbosity 30%. Posttrain: RL. ↗ |
| Nemotron | Nemotron-3-Nano | Multi-environment RLVR (second stage) | posttrain | A second unified multi-environment RLVR run after RLHF. Posttrain: RL. ↗ |
| Nemotron | Nemotron-Cascade | Multi-stage SFT, Stage 1 (16K) | posttrain | General data with parallel thinking/non-thinking responses plus thinking-only math, science, code (R1-0528 / V3-0324 teachers); one epoch. Posttrain: SFT. ↗ |
| Nemotron | Nemotron-Cascade | Multi-stage SFT, Stage 2 (32K) | posttrain | Adds 32K-long math/science/code reasoning, tool-calling and SWE data to the general mix; one epoch. Posttrain: SFT. ↗ |
| Nemotron | Nemotron-Cascade | Cascade RL 1: RLHF | posttrain | GRPO with a 72B reward model; math/competitive-coding prompts excluded; 12K max response, batch 128, 8 rollouts, LR 2e-6, ~800 steps. Posttrain: RL. ↗ |
| Nemotron | Nemotron-Cascade | Cascade RL 2: Instruction-Following RL (IF-RL) | posttrain | Two-stage IF-RL (IFEval then IF-Bench-Train taxonomies) on ~40K filtered Llama-Nemotron IF prompts plus 60K custom; 8K-16K responses. Posttrain: RL. ↗ |
| Nemotron | Nemotron-Cascade | Cascade RL 3: Math RL | posttrain | AceReason-Math problems (paper says 18K, and later 49K filtered to 14K); 8B: 24K→32K→40K, 14B: 28K→40K; temp 1.0, LR 2-2.5e-6, ~100-200 steps per stage. Posttrain: RL. ↗ |
| Nemotron | Nemotron-Cascade | Cascade RL 4: Code RL | posttrain | Single-stage on-policy code RL on the AceReason coding corpus, 44K-48K max response, binary unit-test reward, temp 1.0, LR 4e-6. Posttrain: RL. ↗ |
| Nemotron | Nemotron-Cascade | Cascade RL 5: SWE RL | posttrain | Agentless code-repair RL with an execution-free patch-similarity reward; 16 rollouts, temp 1, 16K max response, LR 2.5e-6. Posttrain: RL. ↗ |
| Nemotron | Nemotron-Terminal | SFT on Terminal-Corpus | posttrain | SFT of Qwen3-8B/14B/32B on DeepSeek-V3.2 terminal trajectories (dataset adapters + Terminal-Task-Gen synthetic tasks); LR 2e-5, 2 epochs, 32,768 tokens. Posttrain: SFT. ↗ |
| Nemotron | Nemotron-3-Super | Pre-training Phase 1 (0-20T tokens) | pretrain | 20T tokens on a diversity-oriented blend; WSD warmup over 200B tokens to 4.5e-4, then stable LR; NVFP4, 8K sequences, batch 3,072 (~25.17M tokens). ↗ |
| Nemotron | Nemotron-3-Super | Pre-training Phase 2 (20T-25T tokens) | pretrain | 5T tokens on a higher-quality blend (finepdfs-high, more stem/code-sft), coinciding with minus-sqrt LR decay to 4.5e-6. Pretrain: data re-weighting and decay inside the main schedule. ↗ |
| Nemotron | Nemotron-3-Super | Long-context phase (LC-Phase), stage 1 | midtrain | Continuous pretraining at 1,048,576 context for 34B tokens, constant LR 4.5e-6, batch 16; 20% document-QA + 80% downscaled Phase 2 data. Midtrain: a named context-extension phase. ↗ |
| Nemotron | Nemotron-3-Super | Long-context phase (LC-Phase), stage 2 | midtrain | 17B tokens alternating 1M and 4K sequences to recover math benchmarks. Midtrain: part of the named context-extension phase. ↗ |
| Nemotron | Nemotron-3-Super | SFT Stage 1 (token-level loss) | posttrain | Over 7M samples with token-level averaged loss; 256K packing, batch 64, constant LR 1e-5 after 30k warmup samples; shared-weight MTP loss (scale 0.3) kept on. ↗ |
| Nemotron | Nemotron-3-Super | SFT Stage 2 (sample-level loss) | posttrain | 85% of the Stage 1 blend plus 256K/512K long-context data; per-conversation normalized loss, 512K packing, batch 32, LR 1e-5; restores long-input-short-output accuracy. ↗ |
| Nemotron | Nemotron-3-Super | Budget-control semi-on-policy SFT | posttrain | 350 steps on the model's own rollouts with 12% of reasoning traces truncated to random budgets, teaching inference-time reasoning budget control. ↗ |
| Nemotron | Nemotron-3-Super | RL Stage 1: Multi-environment RLVR | posttrain | Async GRPO over 21 environments / 37 datasets in rounds (25-30 env types, then an agentic round with 37), 256 prompts x 16 rollouts, max generation 49K->64K, GenRM in the loop, PivotRL for agentic domains. ↗ |
| Nemotron | Nemotron-3-Super | RL Stage 2: SWE-RL | posttrain | End-to-end OpenHands agent loop in Apptainer containers with binary test reward, plus OpenCode/Codex-style tool formats; ~20B tokens (Fig. 12). ↗ |
| Nemotron | Nemotron-3-Super | RL Stage 3: RLHF | posttrain | RLHF-only stage with the principle-following GenRM (initialized from Qwen3-235B-A22B-Thinking-2507); ~18B tokens (Fig. 12). ↗ |
| Nemotron | Nemotron-3-Super | Stage 4: MTP healing | posttrain | Only the MTP heads are trained (NLL on responses generated for RLVR prompts), backbone frozen; significantly improves MTP accuracy. ↗ |
| Nemotron | Nemotron-3-Super | Quantization for inference (PTQ) | posttrain | FP8 PTQ calibrated on 256 SFT samples at 65,536 context; NVFP4 PTQ with MSE-chosen weight scales and AutoQuantize mixed precision (4.75-bit budget), under 2 hours on one 8xB200 node. ↗ |
| Nemotron | Nemotron-3-Nano-4B | Nemotron Elastic compression + Stage 1 short-context distillation (8K) | midtrain | Router-guided pruning of Nemotron Nano 9B v2 to 4B, then KD from the frozen 9B parent on 63B tokens at 8K (~70% post-training, 30% pretraining data). Midtrain: continued training of an earlier base. ↗ |
| Nemotron | Nemotron-3-Nano-4B | Stage 2 long-context extension (49K) | midtrain | Distillation continues for 150B tokens at 49K sequence length. Midtrain: a named context-extension phase. ↗ |
| Nemotron | Nemotron-3-Nano-4B | Supervised Fine-Tuning (two stages) | posttrain | Broad reasoning + non-reasoning SFT (math, code, science, chat, IF, agentic), then a small safety-focused stage. Posttrain: SFT. ↗ |
| Nemotron | Nemotron-3-Nano-4B | Multi-environment Reinforcement Learning (three RLVR stages) | posttrain | NeMo-RL/Gym: single-turn IF, then multi-turn IF + structured outputs, then conversational tool calling. Posttrain: RL. ↗ |
| Nemotron | Cascade-2 | Supervised Fine-Tuning (single stage) | posttrain | SFT of Nemotron-3-Nano-30B-A3B-Base on multi-domain teacher data packed to 256K, ~1.5 epochs (33,000 steps, cosine LR 5e-5 to 5e-6). Posttrain: SFT. ↗ |
| Nemotron | Cascade-2 | Instruction-Following RL (IF-RL) | posttrain | First Cascade RL stage: verifiable IF rewards, thinking mode only, no reward model, dynamic filtering and overlong penalty; ~180 steps, 128 prompts x 16 rollouts, LR 3e-6. Posttrain: RL. ↗ |
| Nemotron | Cascade-2 | Multi-domain RL | posttrain | One joint RL stage on ~55% STEM MCQA, 30% agentic tool calling, 15% structured output; ~70 steps. Posttrain: RL. ↗ |
| Nemotron | Cascade-2 | Multi-domain On-Policy Distillation (MOPD) | posttrain | Token-level reverse-KL on-policy distillation from in-pipeline teachers (SFT ckpt for math, RLHF ckpt, IF-RL + multi-domain ckpt) with truncated importance weights; 52 steps. Posttrain: distillation. ↗ |
| Nemotron | Cascade-2 | Reinforcement Learning from Human Feedback (RLHF) | posttrain | Pairwise GenRM rewards (Qwen3-235B-A22B-Thinking-2507) with length normalization and conciseness bonus, thinking mode only, KL 0.03, 16K responses, ~25-30 steps. Posttrain: RL. ↗ |
| Nemotron | Cascade-2 | Long-context RL | posttrain | Long-context subset of the Nano-v3 RL blend only, Qwen3-235B-A22B-Instruct-2507 as judge, inputs up to 32K, ~30 steps. Posttrain: RL. ↗ |
| Nemotron | Cascade-2 | Code RL | posttrain | 3.5K hard competitive-programming prompts (those GPT-OSS-120B does not solve 8/8), strict binary test-case reward, 118K responses, 22 steps. Posttrain: RL. ↗ |
| Nemotron | Cascade-2 | Software Engineering RL (agentless + execution-based agentic) | posttrain | Agentless code-repair RL with GPT-OSS-120B as reward model (98,304 seq), then OpenHands execution-based RL: 16 prompts x 64 rollouts, SWE-Gym + R2E-Subset. Posttrain: RL. ↗ |
| Nemotron | Nemotron-3-Nano-Omni | SFT Stage 0: Vision projector warmup | midtrain | Vision MLP projector only, 9.35M samples (~15.5B tokens) at 16K. Midtrain: adds vision to a finished LM. ↗ |
| Nemotron | Nemotron-3-Nano-Omni | SFT Stage 1: Vision SFT 16k | posttrain | LLM and C-RADIOv4-H unfrozen on 86.3M vision-language samples (~214.8B tokens) at 16K. Posttrain: lab-named full-model SFT (as Nano V2 VL SFT Stage 1); only projector warm-ups count as midtrain. ↗ |
| Nemotron | Nemotron-3-Nano-Omni | SFT Stage 2: Audio projector warmup | midtrain | Audio projector only on Granary v1.1 ASR, 59.2M samples (~11.4B tokens). Midtrain: adds audio to a finished LM. ↗ |
| Nemotron | Nemotron-3-Nano-Omni | SFT Stage 3: Audio projector & encoder | midtrain | Parakeet encoder + projector on 100.5B tokens (ASR 22.8B, sound 24.5B, music 43.5B, speech 9.6B), LLM frozen. Midtrain: adds audio to a finished LM. ↗ |
| Nemotron | Nemotron-3-Nano-Omni | SFT Stage 4: Omni SFT 16k | posttrain | 30.5M samples (~57.3B tokens) of vision, video, audio, omni, text and safety data, all parameters trainable, 16K. Posttrain: SFT. ↗ |
| Nemotron | Nemotron-3-Nano-Omni | SFT Stage 5: Omni SFT 48k | posttrain | 6.08M samples (~33.5B tokens) weighted to long video, omni and reasoning data at 49,152 context. Posttrain: SFT (context grows inside the SFT curriculum). ↗ |
| Nemotron | Nemotron-3-Nano-Omni | SFT Stage 6: Omni SFT 256k | posttrain | ~34.0B tokens of long-document vision and long-context text at 262,144 context; audio encoder and projector frozen. Posttrain: SFT (context grows inside the SFT curriculum). ↗ |
| Nemotron | Nemotron-3-Nano-Omni | Preference Optimization (MPO) | posttrain | Offline MPO: DPO preference loss + BCO quality loss on rejection-sampled vision responses labeled by correctness. Posttrain: preference optimization. ↗ |
| Nemotron | Nemotron-3-Nano-Omni | Text-RL stage 1, Image-RL, Omni-RL, Text-RL stage 2 | posttrain | Adapted GSPO (batch 4,096, 16 rollouts): text RLVR/RLHF on the LM, image RL with rule verifiers, omni RL on ~120K prompts, then text RL again. Posttrain: RL. ↗ |
| Nemotron | Nemotron-3-Ultra | Pretraining phase 1 (diversity mix, 0-15T tokens) | pretrain | NVFP4 WSD run: 200B-token warmup to 2.5e-4, then stable LR on a diversity-biased 19-category mix (web crawl ~49%) for ~15T tokens. Pretrain: main schedule. ↗ |
| Nemotron | Nemotron-3-Ultra | Pretraining phase 2 (quality mix + LR decay, 15-20T tokens) | pretrain | Quality-biased mix (finepdfs-high upweighted, web ~38%) for the last ~5T with minus-sqrt decay to 2.5e-6; horizon cut to 20T after a ~16T divergence. Pretrain: decay inside the main WSD schedule. ↗ |
| Nemotron | Nemotron-3-Ultra | Long-Context Extension (LC-Phase) | midtrain | CPT from a 500B-window merged checkpoint: 33B tokens, 92% of iterations at 1M and 8% at 4K, 46% long-context + 54% phase-2 data, LR 2.5e-6. Midtrain: a named context-extension phase. ↗ |
| Nemotron | Nemotron-3-Ultra | Supervised Fine Tuning (Stage 1 and Stage 2) | posttrain | Stage 1: 294,912-token packs, GBS 64, 204,800 samples, LR 1.5e-5→1e-6; Stage 2: 515K packs with ≤512K data, 19,200 samples; MTP objective kept. Posttrain: SFT. ↗ |
| Nemotron | Nemotron-3-Ultra | Reinforcement Learning (unified RLVR) | posttrain | Async GRPO over all environments (agentic, code, math, safety, chat, IF, long context), batch 8192 with 16 rollouts, max generation 48K→64K; gives the RLVR student. Posttrain: RL. ↗ |
| Nemotron | Nemotron-3-Ultra | Specialized Teachers | posttrain | SWE, office, search, terminal, tool-use, usability, agentic-safety, chat (GenRM RLHF), IF/factuality, STEM and coding teachers via SFT, PivotRL, GRPO or RLHF. Posttrain. ↗ |
| Nemotron | Nemotron-3-Ultra | MOPD Warmup | posttrain | One-time light SFT of the RLVR student on teacher-distribution data so student rollouts stay in teacher support (GDPval 46.7 with vs 35.3 without). Posttrain: SFT. ↗ |
| Nemotron | Nemotron-3-Ultra | MOPD (two iterations) | posttrain | Async reverse-KL on-policy distillation, PPO-style clipping and IcePop masking, 1,024 prompts × 1 rollout, 192K max generation; round 2 adds teachers branched from MOPD1. Posttrain. ↗ |
| Nemotron | Nemotron-3-Ultra | MTP Boosting | posttrain | Head-only forward-KL (T=2) of the shared MTP head to backbone logits over 7 draft steps, 12K steps at GBS 64 on 8K-token on-policy rollouts. Posttrain. ↗ |
| Nemotron | Nemotron-3-Ultra | Quantization (NVFP4 PTQ) | posttrain | Mixed-precision PTQ at 5.03 bits/element: NVFP4 routed experts, FP8 shared experts and Mamba linears, BF16 attention/latent; FP8 KV cache, FP16-SR SSM cache. Posttrain: quantization. ↗ |
| Nemotron | Nemotron-3.5-Lightning | MTP warmup training | midtrain | 100B tokens training only the new MTP head (standard 2-step recursive) on the frozen Nano 3 Base backbone. Midtrain: continued training of an earlier base. ↗ |
| Nemotron | Nemotron-3.5-Lightning | 2.2 Distillation | midtrain | 1T tokens of logit distillation from Super 3 Base (forward KL 0.9 + CE 0.1, Ultra 3 Phase 2 data); LR 1e-5→1e-4 over 200B, linear decay to 1e-5 over 800B. Midtrain: continued training of an earlier base. ↗ |
| Nemotron | Nemotron-3.5-Lightning | Long-context phase (LC-Phase) | midtrain | 21B tokens at 1M context, constant LR 1e-5, Nemotron 3 Ultra LC blend, on GB200. Midtrain: a named context-extension phase. ↗ |
| Nemotron | Nemotron-3.5-Lightning | Supervised Fine Tuning (single stage) | posttrain | One text-only SFT stage; no vision, no second SFT stage. Posttrain: SFT. ↗ |
| Nemotron | Nemotron-3.5-Lightning | Stage 4: Reinforcement Learning | posttrain | Multi-environment asynchronous GRPO (RLVR) with MTP-accelerated rollouts; no specialist teachers or MOPD. Posttrain: RL. ↗ |
| Nemotron | Nemotron-3.5-Lightning | MTP Boosting | posttrain | 48k steps of MTP-only KD training against the frozen final backbone, LK loss, hidden-state mixing. Posttrain: after SFT/RL. ↗ |
| Nemotron | Nemotron-3.5-Lightning | Post-training quantization + QAD (W4A16 NVFP4) | posttrain | ModelOpt PTQ + quantization-aware distillation to W4A16 NVFP4 (FP8 Mamba projections, BF16 attention); 99.24% median recovery. Posttrain: QAT. ↗ |
| Nemotron | Labs-Competitive-Coding | Supervised fine-tuning on GLM-5.2 traces | posttrain | 1 epoch on 477,642 GLM-5.2 traces (22K problems): AdamW, GBS 64, peak LR 1.5e-5 cosine, 262,144-token packs, TP8 on 128 GB300; no RL or distillation stage. Posttrain: SFT. ↗ |
| Nemotron | Labs-Competitive-Coding | NVFP4 post-training quantization | posttrain | Model Optimizer NVFP4 PTQ for live-contest throughput: 736.8 vs 199.1 tok/s/GPU at 52.8% vs 59.4% IOI 2025 Score@1. Posttrain: quantization. ↗ |
| Nemotron | Bolt-74B | Stage 1: Compression (architecture search + 8K recovery) | midtrain | Star Elastic search + logit KD from frozen Super VL 120B-A12B; router converges in 8B tokens, KD continues to 72B at 8K on 70% parent SFT + 30% Ultra-v3 pretrain data. Midtrain: continued training of an earlier base. ↗ |
| Nemotron | Bolt-74B | Stage 1: Compression (128K recovery) | midtrain | 48B more KD tokens at 128K (128 sequences, ~16.8M tokens per batch) for 120B total; whole compression ~64 h on 128 GB200 GPUs. Midtrain: context extension of a compressed earlier base. ↗ |
| Nemotron | Bolt-74B | Stage 2: Vision Adaptor Training | midtrain | Parent vision tower re-attached; only the projector trained on ~6.6M image-only samples/epoch (captions, VQA, OCR, documents). Midtrain: re-attaching vision to an LM (rule 3), though NVIDIA files it under post-training. ↗ |
| Nemotron | Bolt-74B | Stage 3: Supervised Fine-Tuning (vision-heavy SFT) + SLERP merge | posttrain | Parent SFT blends reweighted to vision (ScreenSpot Pro ~10.1B, WebArena ~9.5B tokens) plus Kimi-K3/GLM 5.2 synthetic text; LLM, adaptor and encoder trained; SLERP 80:20 merge with adaptor-only ckpt. ↗ |
| Nemotron | Bolt-74B | Stage 4: Reinforcement Learning (MCP-Atlas RL, RLHF, SWE-RL) + final merge | posttrain | E2E GRPO on a simulated MCP server (32x16=512) and SWE (64x16=1,024 trajectories/step, 196,608 max tokens, Adam 3e-6); RLHF with Nemotron 3 Ultra GenRM; branches SLERP-merged 75:25. ↗ |
| Nemotron | Bolt-74B | Quantization-Aware Distillation (2 stages) | posttrain | IQ2_XXS experts + NVFP4 Mamba via Jaccard-loss QAD (LR 1e-5, cosine to 1e-7): 26B tokens from an earlier SWE-RL BF16 teacher, then 10M RLHF-blend tokens with the final BF16 teacher. ↗ |
| Nemotron | Nemotron-3.5-Super | MTP warmup training | midtrain | 100B tokens training only the single shared MTP head, with hidden-state mixing, on the frozen Nemotron 3 Super Base. Midtrain: continued training of an earlier base. ↗ |
| Nemotron | Nemotron-3.5-Super | 2.2 Distillation | midtrain | 1T tokens of logit distillation from Ultra 3 Base, Skew Reverse KL α=0.1, LR 4.5e-6→4.5e-5→4.5e-6, load-balance coeff 1e-5. Midtrain: continued training of an earlier base. ↗ |
| Nemotron | Nemotron-3.5-Super | 2.3 Indic Continued Pretraining | midtrain | 100B tokens: 10B synthetic + natural Indic, 90B prior mixture; Indic benchmarks +4.98 to +9.82 pt. Midtrain: named continued pre-training. ↗ |
| Nemotron | Nemotron-3.5-Super | Long-context phase (LC-Phase) | midtrain | 35B tokens CPT at 1M context, constant LR 4.5e-6, Ultra 3 LC blend, GB300 (CP16/TP8/EP64). Midtrain: a named context-extension phase. ↗ |
| Nemotron | Nemotron-3.5-Super | 2.5 Mid-training (agentic software engineering) | midtrain | 65.4B tokens at 256K of issue-PR, repos, commits, CI logs, 10% agent traces; LR re-warmed to 1e-5 then constant. Midtrain: named mid-training stage. ↗ |
| Nemotron | Nemotron-3.5-Super | Long-context recovery | midtrain | 9B tokens with the LC-Phase recipe after 256K mid-training; RULER-1M back from 33.5 to 81.0. Midtrain: context-length stage after the main run. ↗ |
| Nemotron | Nemotron-3.5-Super | Vision projector warm-up + two-stage SFT | posttrain | Text-only base becomes a VLM: stage 1 broad vision/reasoning/chat/long-context/safety/agentic mix, small rebalanced stage 2; 524,288-token packing, batch 32, LR 1e-5. Posttrain: SFT. ↗ |
| Nemotron | Nemotron-3.5-Super | 3.2 Reinforcement Learning (RLVR + RLHF) | posttrain | Async GRPO, batch 8,192 x 16 rollouts, generation 80K→128K, router replay + FP32 LM head; step-120 checkpoint is the trunk for teachers and MOPD student. Posttrain: RL. ↗ |
| Nemotron | Nemotron-3.5-Super | Specialized teachers (agentic, reasoning, vision, video, alignment) | posttrain | Teachers branched from RL step 120 via async GRPO (agentic sandboxes, 131K-token reasoning, video QA + SA-V tracking, vision tools, vision RLVR) and RLHF (alignment). Posttrain: RL. ↗ |
| Nemotron | Nemotron-3.5-Super | 3.6 MOPD (two stages) | posttrain | On-policy reverse-KL distillation from specialists into RL step-120 student: stage 1 30 steps (LR 2e-6), stage 2 40 steps (LR 3e-6, α=0.2 proximal teacher). Posttrain: distillation. ↗ |
| Nemotron | Nemotron-3.5-Super | 3.7 MTP Boosting | posttrain | Frozen backbone; MTP head trained 4k steps at SFT, then 25k steps on the final backbone with LK-loss KD; AL at temp 1 3.787→4.612. Posttrain: after RL/MOPD. ↗ |
| Nemotron | Nemotron-3.5-Super | 4. Quantization (PTQ + QAD) | posttrain | Aggressive NVFP4 PTQ (MSE calibration on multimodal SFT data) then 200-iteration SFT-based QAD with a KL objective; 4.82 bits/param, 99.62% median recovery. Posttrain: QAT. ↗ |
| Mistral | Small-3 | Pretrained base checkpoint (Mistral-Small-24B-Base-2501) | pretrain | 24B dense base, 40 layers, 32k context, trained by Mistral; tokens, data mix, schedule and hardware not disclosed. ↗ |
| Mistral | Small-3 | Instruction tuning (no RL, no synthetic data) | posttrain | Instruction fine-tuning only; the lab states that neither RL nor synthetic data was used. ↗ |
| Mistral | Small-3.1 | Building upon Mistral Small 3: vision and 128k context | midtrain | Small 3 is trained further with a 24-layer Pixtral-type vision encoder and its context goes from 32k to 128k (rope_theta 1e8 -> 1e9); tokens not given. Midtrain: continued training of an earlier base that adds vision. ↗ |
| Mistral | Small-3.1 | Instruction fine-tuning | posttrain | Instruct model fine-tuned from the 3.1 base; data and methods not described. ↗ |
| Mistral | Devstral-Small-1.0 | Long context extension phase (base model) | midtrain | The Mistral Small base (40 layers, GQA) gets a long-context extension to 128k before agentic fine-tuning; no token counts. Midtrain: a named context-extension phase. ↗ |
| Mistral | Devstral-Small-1.0 | Two-stage SFT on agent trajectories | posttrain | SFT on OpenHands CodeAct trajectories from SWE-Gym: first a broad heuristic-filtered set, then only trajectories passing the strictest filters; vision encoder removed. ↗ |
| Mistral | Devstral-Small-1.0 | Policy optimization on fresh rollouts | posttrain | Further rounds of rollouts from the SFT model, then training with an unnamed policy-optimization method. ↗ |
| Mistral | Magistral-Small-1.0 | Reasoning SFT cold start | posttrain | 4 epochs of SFT on Magistral Medium traces (RL-run traces plus answers to OpenThoughts/OpenR1-code prompts) with 10% general instruction data; best AIME'24 checkpoint kept. ↗ |
| Mistral | Magistral-Small-1.0 | RL on top of reasoning SFT bootstrapping | posttrain | Modified GRPO (no KL, eps_high 0.3, temperature 1.0), batch 2048, 32k non-penalized length, rule-based math/code rewards plus format, length and language terms. ↗ |
| Mistral | Small-3.2 | Minor instruct update | posttrain | Further post-training of Small 3.1 Instruct for instruction following, fewer infinite generations and a more robust function-calling template; method not described. ↗ |
| Mistral | Devstral-Small-1.1 | Refreshing Devstral-Small data | posttrain | Same 24B model retrained on refreshed agent data: pseudo-scaffolds, XML and native function-calling prompts, recalibrated first-stage trajectory filtering. ↗ |
| Mistral | Voxtral | Pretraining: adapter warm-up | midtrain | First pass trains only the audio-to-LM adapter with the Whisper encoder and Mistral Small 3.1 decoder frozen. Midtrain: continued training of an earlier base to add audio. ↗ |
| Mistral | Voxtral | Pretraining: audio-text patterns | midtrain | Audio-to-text repetition and cross-modal continuation samples (marked <repeat>/<next>) balanced 50/50, plus text pretraining data. Midtrain: continued training of an earlier base to add audio. ↗ |
| Mistral | Voxtral | Supervised finetuning | posttrain | Mostly synthetic SFT: audio QA, summaries and translations from Mistral Large, TTS-converted text SFT incl. function calling, real-speech questions, transcribe mode. ↗ |
| Mistral | Voxtral | Preference alignment (DPO and Online DPO) | posttrain | DPO then Online DPO: two samples at T=0.5 ranked by a text reward model on the transcribed conversation; the online variant helped more. ↗ |
| Mistral | Magistral-Small-1.2 | SFT from Magistral Medium traces and RL on top | posttrain | Same SFT-then-RL recipe, now starting from Small 3.2 and keeping a vision encoder; no further details. ↗ |
| Mistral | Large-3 | Pretraining from scratch on 3000 H200s | pretrain | 675B-A41B granular MoE with a 2.5B vision encoder trained from scratch on 3000 H200s; tokens, data and schedule not disclosed; YaRN factor 36 from 8,192 in params.json. ↗ |
| Mistral | Large-3 | Instruct post-training | posttrain | Instruction post-training to a non-reasoning instruct model released in FP8; methods not disclosed. ↗ |
| Mistral | Ministral-3 | Cascade Distillation: prune + short-context distillation | midtrain | Prune Small 3.1 (or the previous child), then forward-KL distillation from Small 3.1 on text and image-text at 16k; 1-3T tokens per model. Midtrain: continued training of an earlier base (lab calls it pretraining). ↗ |
| Mistral | Ministral-3 | Long context stage | midtrain | Further distillation from Small 3.1 extending 16,384 -> 262,144 with YaRN and position-based temperature scaling. Midtrain: a context-extension phase. ↗ |
| Mistral | Ministral-3 | Post-Training: Ministral Instruct (SFT + ODPO) | posttrain | fp8 SFT with logit distillation from Mistral Medium 3, then Online DPO with a pairwise reward model, loop-penalizing heuristics and tool execution. ↗ |
| Mistral | Ministral-3 | Post-Training: Ministral Reasoning (SFT, GRPO, ODPO) | posttrain | Short+long CoT SFT (3B distilled from Magistral Small 1.2), GRPO in STEM then rubric-judged General stages (32K -> 80K generation), then ODPO on thinking-stripped outputs. ↗ |
| Mistral | Devstral-2 | Instruct fine-tuning (FP8) | posttrain | Agentic-coding instruct fine-tuning of a 123B dense model (and a 24B one from Small 3.1 base); data and methods not disclosed. ↗ |
| Mistral | Leanstral | Proof-engineering specialization | posttrain | A Small 4-family MoE specialized for Lean 4 proof engineering in realistic repositories and lean-lsp-mcp; training method deferred to a tech report. ↗ |
| Mistral | Leanstral-1.5 | Mid-training | midtrain | First of three stages; data and token counts not described. Midtrain: a phase the lab names mid-training. ↗ |
| Mistral | Leanstral-1.5 | Supervised fine-tuning | posttrain | SFT on an improved Lean proof-engineering mixture; details not given. ↗ |
| Mistral | Leanstral-1.5 | Reinforcement learning with CISPO | posttrain | CISPO RL in a multiturn prove-or-disprove environment with compiler feedback and a code-agent environment (bash, Lean LSP, compaction) verified by a SafeVerify fork. ↗ |
| Mistral | Large-4 | Training from scratch on Grace Blackwell | pretrain | ~1T-A52B multimodal granular MoE trained from scratch on 3,800 GB-series GPUs, 160+ languages in the data; tokens and schedule not yet disclosed. ↗ |
| Mistral | Large-4 | Post-training: SFT and RL at scale | posttrain | SFT, then asynchronous RL on ~3k GPUs (~33B tokens/day, ~16B trainable) over composable environments; the RL run continues after the preview. ↗ |
