Behavior Injection: Preparing Language Models for Reinforcement Learning

2025/05/25 by Zhepeng Cen, Cen, Zhepeng, Yihang Yao +7 · 3 citations
Computer Science · #Artificial Intelligence (cs.AI) #Evolutionary Algorithms and Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Reinforcement Learning in Robotics #Software Engineering Research

paper · pdf · doi:10.48550/arxiv.2505.18917

openalex publication_date 2025/05/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Reinforcement learning (RL) has emerged as a powerful post-training technique to incentivize the reasoning ability of large language models (LLMs). However, LLMs can respond very inconsistently to RL finetuning: some show substantial performance gains, while others plateau or even degrade. To understand this divergence, we analyze the per-step influence of the RL objective and identify two key conditions for effective post-training: (1) RL-informative rollout accuracy, and (2) strong data co-influence, which quantifies how much the training data affects performance on other samples. Guided by these insights, we propose behavior injection, a task-agnostic data augmentation scheme applied prior to RL. Behavior injection enriches the supervised finetuning (SFT) data by seeding exploratory and exploitative behaviors, effectively making the model more RL-ready. We evaluate our method across two reasoning benchmarks with multiple base models. The results demonstrate that our theoretically motivated augmentation can significantly increase the performance gain from RL over the pre-RL model.

Citations

• SWE-smith: Scaling Data for Software Engineering Agents
• RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
• ToolRL: Reward is All Tool Learning Needs
• Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
• APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay
• Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving
• GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning
• Understanding R1-Zero-Like Training: A Critical Perspective
• SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild
• What Makes a Reward Model a Good Teacher? An Optimization Perspective
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Models
• Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
• Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning
• R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning
• All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuning
• Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs
• MedVLM-R1: Incentivizing Medical Reasoning Capability of Vision-Language Models (VLMs) via Reinforcement Learning
• SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution
• Your Language Model May Think Too Rigidly: Achieving Reasoning Consistency with Symmetry-Enhanced Training
• From System 1 to System 2: A Survey of Reasoning Large Language Models
• Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning
• Magma: A Foundation Model for Multimodal AI Agents
• Demystifying Long Chain-of-Thought Reasoning in LLMs
• Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search
• Lower Bounds for Chain-of-Thought Reasoning in Hard-Attention Transformers
• Process Reinforcement through Implicit Rewards
• Reinforcement Learning for Long-Horizon Interactive LLM Agents
• IntellAgent: A Multi-Agent Framework for Evaluating Conversational AI Systems
• Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thought
• Qwen2.5 Technical Report
• MAG-V: A Multi-Agent Framework for Synthetic Data Generation and Verification
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training
• What Do Learning Dynamics Reveal About Generalization in LLM Reasoning?
• Language Models are Hidden Reasoners: Unlocking Latent Reasoning Capabilities via Self-Rewarding
• WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learning
• On Memorization of Large Language Models in Logical Reasoning
• Synthesizing Post-Training Data for LLMs through Multi-Agent Simulation
• Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization
• Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents
• Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems
• Agent Q: Advanced Reasoning and Learning for Autonomous AI Agents
• The Llama 3 Herd of Models
• Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Process
• AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents
• Recursive Introspection: Teaching Language Model Agents How to Self-Improve
• Learning Dynamics of LLM Finetuning
• APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets
• DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning
• AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents
• OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems
• Dynamic Evaluation of Large Language Models by Meta Probing Agents
• Chain of Thought Empowers Transformers to Solve Inherently Serial Problems
• OpenMathInstruct-1: A 1.8 Million Math Instruction Tuning Dataset
• Premise Order Matters in Reasoning with Large Language Models
• LESS: Selecting Influential Data for Targeted Instruction Tuning
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Augmenting Math Word Problems via Iterative Question Composing
• ReFT: Reasoning with Reinforced Fine-Tuning
• PromptBench: A Unified Library for Evaluation of Large Language Models
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• TAIL: Task-specific Adapters for Imitation Learning with Large Pretrained Models
• DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
• MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Autonomous Tester Agent Benchmark
• Android in the Wild: A Large-Scale Dataset for Android Device Control
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective
• Physics of Language Models: Part 1, Learning Hierarchical Language Structures
• GPT-4 Technical Report
• LLaMA: Open and Efficient Foundation Language Models
• A Kernel-Based View of Language Model Fine-Tuning
• STaR: Bootstrapping Reasoning With Reasoning
• Training language models to follow instructions with human feedback
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Training Verifiers to Solve Math Word Problems
• Evaluating Large Language Models Trained on Code
• Measuring Mathematical Problem Solving With the MATH Dataset
• Conservative Q-Learning for Offline Reinforcement Learning
• Language Models are Few-Shot Learners
• Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
• Estimating Training Data Influence by Tracing Gradient Descent
• Neural Tangent Kernel: Convergence and Generalization in Neural Networks
• Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor
• Proximal Policy Optimization Algorithms
• Deep Exploration via Bootstrapped DQN
• Trust Region Policy Optimization
• Do Theory of Mind Benchmarks Need Explicit Human-like Reasoning in Language Models?
• The Expressive Power of Transformers with Chain of Thought
• OpenAI o1 System Card

Cited by

• Tailored Primitive Initialization is the Secret Key to Reinforcement Learning
• Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead
• Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levels

Related

• ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
• Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
• Representation-Based Exploration for Language Models: From Test-Time to Post-Training
• Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection
• Towards Revealing the Effectiveness of Small-Scale Fine-tuning in R1-style Reinforcement Learning