DeepSeek-R1: RL-trained reasoning and distillation

Feb 8, 2025

What DeepSeek-R1 is optimizing for

A model may be good at math, code, or logic puzzles without guaranteeing correctness. Supervised fine-tuning (SFT) can teach a large language model (LLM) to imitate worked solutions; reinforcement learning (RL) can instead update it using rewards for its own responses. DeepSeek’s experiment asks how far reasoning can improve when that post-training starts directly from a pretrained model, without an initial SFT stage. This article covers the original January 2025 models and technical report.

By early 2025, OpenAI had released reasoning models in the o1 series and o3-mini. DeepSeek published a training approach and model weights that let researchers examine another route to similar capabilities.

DeepSeek-R1 and DeepSeek-R1-Zero both use Group Relative Policy Optimization (GRPO), which avoids training a separate value network. Their reasoning training uses automated correctness feedback, with additional rewards for output format or language consistency. R1’s later training also uses learned preference rewards, and its training data is used to fine-tune smaller models.

R1-Zero uses pure RL for post-training, while R1 adds supervised fine-tuning and preference alignment to improve readability and general usefulness.

R1-Zero and R1: architecture and training

Both models use DeepSeek-V3-Base, a 671B-parameter Mixture-of-Experts (MoE) transformer. Only ~37B parameters are active for any given token, so gating increases capacity without a proportional increase in compute. The original releases support a 128K context window, which gives them room for long chain-of-thought (CoT) responses. These are the released models’ specifications, not a claim that every evaluation used the full context.

Their parent model, DeepSeek-V3, was pretrained with an auxiliary Multi-Token Prediction (MTP) objective: it predicted one additional future token alongside the usual next token. This is a feature of V3’s training, not an extra reasoning reward in the R1 pipeline. The main model can run without the MTP module; that module can also be used for speculative decoding. The V3 report describes both options.

R1-Zero applies RL directly to the pretrained base, without first fine-tuning on example responses. It still needs training questions and correctness checks; skipping SFT does not mean using no data. Its rewards check answer accuracy and reasoning-tag format, without a learned outcome or process reward model. The authors report strong reasoning benchmark results alongside poor readability and language mixing. They reserve the cold-start-data stage for R1. R1 report, sections 2.2–2.3.

R1 starts separately from DeepSeek-V3-Base, using a small supervised cold-start set, two RL stages, another SFT stage, and rejection sampling to construct training data. These additions improve reasoning and readability, though they do not eliminate language mixing: the original model is optimized for Chinese and English and can mix languages when prompted in others. R1 report, sections 2.3 and 5.

Training R1-Zero with pure RL

The authors took DeepSeek-V3-Base, pretrained on ~14.8T tokens, and applied RL without a preceding SFT stage. That claim concerns post-training; it does not mean the model had never encountered worked solutions during pretraining. The R1 reward specification describes:

Accuracy rewards based on the final answer, such as checking a math result or running code against predefined tests.
No process-level reward model: they verify the final answer rather than checking each chain-of-thought step.
A separate format reward for placing reasoning between <think> and </think> tags.

The training template explicitly asks for reasoning followed by an answer. It does not prescribe reflection, backtracking, or a particular solution strategy. This leaves room for RL to reinforce strategies that score well, while the format reward still constrains how responses are presented. R1 training template.

If you’re unfamiliar with Supervised Fine-Tuning (SFT): it’s training on prompt → ideal completion examples.

Supervised fine-tuning with instruction and output examples

The illustration shows instruction–response training examples. Its word-level description is simplified: these models predict tokens, which need not be whole words.

Ground-truth rewards

A reward model scores a response; a value network, often called the critic, estimates expected future reward during the policy update. Those are different jobs. R1-Zero avoids learned reward models because the authors observed reward hacking and wanted to avoid the cost of retraining them. Separately, GRPO removes the value network by estimating a baseline from a group of response scores. R1 reward design, DeepSeekMath’s PPO–GRPO comparison.

Rule-based checks are useful when correctness can be evaluated reliably, but passing a finite test suite is not a proof that a program is correct on every input. Removing a learned reward model does not make the reward specification impossible to exploit.

Emergent self-reflection and “Aha Moments”

The authors observed longer responses, self-checking, and reconsideration during RL, without a separate set of SFT demonstrations teaching those behaviors. Their training curves show response length increasing alongside benchmark performance. A simplified account of the update is:

The model samples several responses to a question.
Each receives accuracy and format rewards.
GRPO compares those rewards within the group.
The update favors above-average responses and discourages below-average ones, subject to clipping and regularization.

Longer reasoning can become more likely when it earns higher relative reward. Length itself is not the reward, and an incorrect answer does not necessarily receive zero total reward because the format component is separate. The paper’s “aha moment” is an example of an intermediate checkpoint reconsidering its approach, not proof that the model’s underlying reasoning was absent before RL. R1 report, section 2.2.4.

R1-Zero’s strong benchmark performance came with poor readability and language mixing. The report does not establish that particular fragments of code or mathematical notation appeared because they had individually been rewarded.

Group Relative Policy Optimization (GRPO)

Why GRPO instead of PPO or value models?

PPO-based RL for language models often uses a value network similar in size to the policy. Training another large network adds memory and compute costs. GRPO, introduced in DeepSeekMath, replaces that learned baseline with statistics from several responses to the same question. Jay Alammar illustrates the process:

Sampling: For each prompt, the model generates multiple responses (“a group”).
Reward Calculation: Each response receives the task’s rewards, including correctness and format checks for R1-Zero. The diagrams use an illustrative coding task.
Relative Advantages: Subtract the group’s mean reward and divide by its standard deviation. When the rewards differ, above-average responses have positive advantages and below-average responses have negative ones; identical scores provide no relative ranking signal.
Update: Use a clipped PPO-like policy objective with these advantages and a KL-divergence penalty relative to a reference policy. There is no learned value baseline. The published R1 objective is shown above.

Removing the value network reduces memory overhead. It does not remove reward hacking: GRPO can use learned reward models, as DeepSeekMath itself does, and its behavior still depends on what the reward function measures.

How GRPO favors longer reasoning

When longer chain-of-thought solutions earn higher rewards than other responses to the same question, GRPO can favor them. A short correct response can also score well; the objective does not directly prefer longer text. The increased response lengths and self-checking in R1-Zero are observations from that training run, not guarantees of the algorithm.

The additional training phases for DeepSeek-R1

R1 adds supervised data and broader alignment training to address R1-Zero’s readability problems. The report describes four stages: two SFT stages and two RL stages. Below, dataset generation and the second SFT stage are shown separately, following Maarten Grootendorst’s illustrations. Their numbered intermediate checkpoint names and reward scores are explanatory labels, not a published training specification.

Phase 1: Small “Cold-Start” supervised fine-tuning

A purely RL-trained model can produce unreadable or disorganized text. R1 first fine-tunes DeepSeek-V3-Base on thousands of long-CoT examples, collected through prompting, readable R1-Zero outputs, and human post-processing. The examples use a readable reasoning-and-summary format to give RL a more stable starting point. They reduce readability problems rather than guaranteeing that language mixing disappears. Cold-start procedure.

The illustration’s ≈5,000 samples is more specific than the report, which states only that there were thousands of examples.

Phase 2: RL for reasoning accuracy

After the cold start, the model trains with the same large-scale RL approach used for R1-Zero, on rule-verifiable math, coding, science, and logic problems:

Generate candidate solutions.
Score accuracy and use GRPO to compare responses within each group.
Add a language-consistency reward, measured as the proportion of target-language words in the CoT.

The report describes summing accuracy and language-consistency rewards. The latter improves readability but slightly reduces performance in the authors’ ablation, so it is a trade-off rather than a free improvement. Reasoning-oriented RL.

The numeric rewards and separate <answer>-tag score in the illustration are illustrative, not published reward values or a complete scoring rule.

Phase 3: Building a verified dataset through rejection sampling

Once reasoning-oriented RL converged, the authors used that checkpoint to construct the reasoning portion of a new SFT dataset:

Pose fresh or more diverse problems.
Collect multiple candidate solutions.
Filter for correct answers using rule-based checks and, for some additional problems, DeepSeek-V3 as a judge given the ground truth and model prediction.
Filter out unreadable traces, including mixed languages, long paragraphs, and code blocks.

This produced about 600,000 reasoning samples. Correctness filtering checks the answer; it does not certify every intermediate reasoning step. About 200,000 general-purpose samples came from the DeepSeek-V3 pipeline, including reused V3 SFT data and some V3-generated answers. Together, these form roughly 800,000 examples. Dataset construction.

The diagram labels the judge and general-data generator as DeepSeek-V3-Base. The report identifies DeepSeek-V3 for those roles and also includes reused SFT data.

Phase 4: Second supervised fine-tuning

The authors return to DeepSeek-V3-Base and fine-tune it for two epochs on the roughly 800k-sample dataset. This trains it to imitate the selected reasoning and general-purpose responses. SFT does not make generation deterministic, and the report does not isolate a guaranteed improvement in every aspect of readability or accuracy from this stage alone. Second SFT stage.

The checkpoint shown at the bottom still undergoes the final RL stage before becoming the released DeepSeek-R1.

Phase 5: Final RL stage with preference alignment

The final RL stage mixes reasoning tasks with general-purpose prompts. Rule-based rewards remain in use for reasoning; learned reward models capture human preferences on the general tasks. Helpfulness is assessed on the final summary, while harmlessness is assessed on the entire response, including the reasoning. The summary format was already introduced in the cold-start data. RL for all scenarios.

Process-level and outcome-level reward models

A process reward model (PRM) scores intermediate reasoning steps, while an outcome reward model (ORM) scores the completed response. Let’s Verify Step by Step studies that distinction. A separate approach, LaTent Reasoning Optimization (LaTRO), treats reasoning as a latent variable and optimizes it using the model’s own answer-likelihood signal; it does not require an external step-scoring reward model.

R1-Zero uses rule-based outcome and format rewards without a neural PRM or ORM. R1 later introduces model-based judgments during data filtering and preference training. Rewarding correct final answers can encourage useful intermediate work, but the final answer alone does not verify the reasoning trace.

Emergent “Self-Reflection” and exploration

The R1-Zero report describes self-verification and alternative solution attempts appearing during RL without explicit instructions to use those strategies. That is evidence about the observed training run. It does not establish that longer reasoning always catches more errors, or that an apparently reflective passage faithfully explains the model’s computation. R1 also receives supervised cold-start examples, so its post-training does not isolate RL’s effects from SFT in the same way. R1-Zero observations, R1 cold-start data.

Distillation into smaller models

Training or serving a 671B MoE model requires substantial resources. DeepSeek released six dense distilled models: Qwen-based 1.5B, 7B, 14B, and 32B models, plus Llama-based 8B and 70B models. Their backbones are Qwen2.5-Math-1.5B/7B, Qwen2.5-14B/32B, Llama-3.1-8B, and Llama-3.3-70B-Instruct. The official model list records the exact pairings.

Data: The report uses the same roughly 800,000 curated samples described above, combining filtered reasoning responses from the R1 pipeline with general-purpose data.
Student: A smaller Qwen or Llama model is fine-tuned to reproduce those response sequences. The released distilled models use SFT only, without an additional RL stage.

This is sequence-based distillation, not a requirement to match the teacher’s full token-probability distribution. It is also not direct distillation from R1-Zero: the report identifies the R1 pipeline and points to its curated SFT dataset. Distillation procedure.

Distillation is supervised imitation. It can transfer useful reasoning behaviors and materially improve benchmark performance, but it doesn’t give you the same training signal as RL on verifiable tasks, so generalization depends on the student model and the distillation data.

The results depend on the benchmark. In the original evaluation, the 7B student scored 55.5% on AIME 2024, above QwQ-32B-Preview’s 50.0%, but scored lower on LiveCodeBench (37.6% versus 41.9%). The 70B student reached 94.5% on MATH-500 versus R1’s 97.3%, while its AIME pass@1 was 70.0% versus R1’s 79.8%. These are DeepSeek’s reported results under its evaluation setup, not a general equivalence between the models. Original benchmark tables, evaluation and sampling settings.

Table 5 from the original R1 report. Pass@1 estimates single-response accuracy; cons@64 uses majority voting across 64 responses. The two columns measure different inference procedures.

Limits of MCTS, process reward models, and small-model RL

The authors’ Monte Carlo Tree Search (MCTS) experiments encountered a large token search space, local optima when expansions were capped, and difficulty training a reliable value model. They still found that MCTS could improve inference with a pretrained value model; the unsuccessful part was repeatedly improving the model through that search-and-training loop. Their PRM experiments ran into problems defining and labeling reasoning steps, reward hacking, and retraining overhead. These are limitations of those experiments, not evidence that either approach cannot work. Unsuccessful attempts.

In a separate comparison, Qwen-32B-Base trained with RL for over 10,000 steps underperformed the distilled Qwen-32B on the reported benchmarks. That supports distillation for this setup; it does not establish a general limit on what small models can learn through RL. Distillation versus RL.

GRPO reduces the cost of large-scale RL by removing a separate value network. Reliable verifiers make outcome rewards easier to define, but the reward still measures only what the verifier checks. For open-ended tasks, R1’s own pipeline continues to rely on preference data and learned reward models.