vix.ing · top · new · best · stats · spec

DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning

DeepSeek-R1 shows an LLM can learn strong step-by-step reasoning from pure reinforcement learning, with no human-labeled reasoning examples.

2025/01/22 by DeepSeek-AI, Daya Guo, Guo, Daya +404 · 93 voices · 2191 citations
Computer Science · #Reinforcement Learning in Robotics #Data Stream Mining Techniques #Explainable Artificial Intelligence (XAI)

paper · pdf · doi:10.1038/s41586-025-09422-z

Abstract

Abstract General reasoning represents a long-standing and formidable challenge in artificial intelligence (AI). Recent breakthroughs, exemplified by large language models (LLMs) 1,2 and chain-of-thought (CoT) prompting 3 , have achieved considerable success on foundational reasoning tasks. However, this success is heavily contingent on extensive human-annotated demonstrations and the capabilities of models are still insufficient for more complex problems. Here we show that the reasoning abilities of LLMs can be incentivized through pure reinforcement learning (RL), obviating the need for human-labelled reasoning trajectories. The proposed RL framework facilitates the emergent development of advanced reasoning patterns, such as self-reflection, verification and dynamic strategy adaptation. Consequently, the trained model achieves superior performance on verifiable tasks such as mathematics, coding competitions and STEM fields, surpassing its counterparts trained through conventional supervised learning on human demonstrations. Moreover, the emergent reasoning patterns exhibited by these large-scale models can be systematically used to guide and enhance the reasoning capabilities of smaller models.

Summary

The authors train DeepSeek-V3-Base with reinforcement learning alone, rewarding only correct final answers and proper formatting, and find the model teaches itself long chain-of-thought reasoning, jumping from 15.6% to 77.9% pass@1 on the AIME 2024 math competition. Because this pure-RL model (DeepSeek-R1-Zero) is hard to read and mixes languages, they add a small set of human-curated examples plus further RL and fine-tuning stages to produce DeepSeek-R1, which keeps the reasoning gains while also handling writing and general instructions, and which they distill into smaller models.

machine-generated · claude-sonnet-5

Outline

machine-generated · claude-sonnet-5

Claims

machine-generated · claude-sonnet-5

Key figure

Figure 1 — Two charts tracking DeepSeek-R1-Zero during pure reinforcement-learning training: its accuracy on the AIME 2024 math competition climbs from 15.6% to 77.9% (86.7% with majority-vote decoding), while its average response length steadily grows, showing the model teaching itself to 'think longer' on harder problems without being told to.

machine-generated · claude-sonnet-5

Glossary

Reinforcement learning (RL)
Training a model by rewarding good outputs and penalizing bad ones, rather than showing it labeled correct answers directly.
Group Relative Policy Optimization (GRPO)
An RL algorithm that scores each of several sampled outputs to the same question against each other, avoiding the need for a separately trained value model that standard PPO requires.
Chain-of-thought (CoT)
The step-by-step reasoning text a model writes out before giving its final answer.
Cold-start data
A small set of human-curated, readable reasoning examples used to warm-start training before large-scale RL begins.
Rejection sampling
Generating many candidate outputs, keeping only the correct or high-quality ones, and using those as training data.
Distillation
Training a smaller model to reproduce the outputs of a larger, more capable model.
Reward hacking
When a model finds a way to score well on its reward signal without actually doing what that reward was meant to measure.
Pass@1
The percentage of problems a model solves correctly on its first attempt.

machine-generated · claude-sonnet-5

Audience

ML researchers and engineers working on LLM post-training, reasoning, or RL fine-tuning, and infrastructure engineers building large-scale RL training systems.

prerequisites: Familiarity with LLM pretraining, supervised fine-tuning, and RLHF pipelines, Basic policy-gradient RL concepts such as PPO, KL regularization, and advantage estimation, Familiarity with reasoning benchmarks like AIME, MATH, Codeforces, and MMLU

machine-generated · claude-sonnet-5

Open questions

machine-generated · claude-sonnet-5

Supplementary links

machine-generated · claude-sonnet-5

Citations

Cited by

Discussions

Related