Reinforcement Pre-Training

2025/06/09 by Quennie Dong, Qingxiu Dong, Li Dong +13 · 5 voices · 14 citations
Computer Science · #Topic Modeling #Natural Language Processing Techniques #Text Readability and Simplification

paper · pdf · doi:10.48550/arxiv.2506.08007

Abstract

In this work, we introduce Reinforcement Pre-Training (RPT) as a new scaling paradigm for large language models and reinforcement learning (RL). Specifically, we reframe next-token prediction as a reasoning task trained using RL, where it receives verifiable rewards for correctly predicting the next token for a given context. RPT offers a scalable method to leverage vast amounts of text data for general-purpose RL, rather than relying on domain-specific annotated answers. By incentivizing the capability of next-token reasoning, RPT significantly improves the language modeling accuracy of predicting the next tokens. Moreover, RPT provides a strong pre-trained foundation for further reinforcement fine-tuning. The scaling curves show that increased training compute consistently improves the next-token prediction accuracy. The results position RPT as an effective and promising scaling paradigm to advance language model pre-training.

Citations

• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking
• Training Compute-Optimal Large Language Models
• Training language models to follow instructions with human feedback

Cited by

• Diversity or Precision? A Deep Dive into Next Token Prediction
• DiG: Differential Grounding for Enhancing Fine-Grained Perception in Multimodal Large Language Model
• PretrainZero: Reinforcement Active Pretraining
• Refine Thought: A Test-Time Inference Method for Embedding Model Reasoning
• Catch Your Breath: Adaptive Computation for Self-Paced Sequence Production
• Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levels
• Agentic Jigsaw Interaction Learning for Enhancing Visual Perception and Reasoning in Vision-Language Models
• Learning to Reason as Action Abstractions with Scalable Mid-Training RL
• Reinforcement Mid-Training
• RLP: Reinforcement as a Pretraining Objective
• Thinking Augmented Pre-training
• Reinforcement Learning on Pre-Training Data
• Advancing Event Forecasting through Massive Training of Large Language Models: Challenges, Solutions, and Broader Impacts
• Shop-R1: Rewarding LLMs to Simulate Human Behavior in Online Shopping via Reinforcement Learning

Discussions

• Reinforcement Pre-Training [hn, 70 points, 18 comments]
• What if the cherry was actually cake? (source: arxiv.org/abs/2506.08007) [bsky, 5 points, 1 comments]
• Reinforcement Pre-Training https://arxiv.org/abs/2506.08007 [bsky, 0 points, 0 comments]
• Reinforcement Pre-Training https://arxiv.org/abs/2506.08007 [comments] [34 points] [bsky, 0 points, 0 comments]
• 8/8 🔮 Co dalej: To dopiero początek. Zespół planuje skalowanie na większe modele i ogólne dane internetowe. RPT może stać się nowym standardem trenowania AI - od "cherry on top" do "cherry cake" 🍰 L [bsky, 0 points, 0 comments]

Related

• Reinforcement Learning on Pre-Training Data
• RLP: Reinforcement as a Pretraining Objective
• Reinforcement Mid-Training
• Diversity or Precision? A Deep Dive into Next Token Prediction
• BOW: Reinforcement Learning for Bottlenecked Next Word Prediction