Enhancing Math Reasoning in Small-sized LLMs via Preview Difficulty-Aware Intervention

2025/08/03 by Xinhan Di, Di, Xinhan, JoyJiaoW · 2 citations
Computer Science · #Reinforcement Learning in Robotics #Topic Modeling #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2508.01604

Abstract

Reinforcement learning scaling enhances the reasoning capabilities of large language models, with reinforcement learning serving as the key technique to draw out complex reasoning. However, key technical details of state-of-the-art reasoning LLMs, such as those in the OpenAI O series, Claude 3 series, DeepMind's Gemini 2.5 series, and Grok 3 series, remain undisclosed, making it difficult for the research community to replicate their reinforcement learning training results. Therefore, we start our study from an Early Preview Reinforcement Learning (EPRLI) algorithm built on the open-source GRPO framework, incorporating difficulty-aware intervention for math problems. Applied to a 1.5B-parameter LLM, our method achieves 50.0% on AIME24, 89.2% on Math500, 77.1% on AMC, 35.3% on Minerva, and 51.9% on OBench, superpass O1-Preview and is comparable to O1-mini within standard school-lab settings.

Citations

• Qwen3 Technical Report
• Process Reinforcement through Implicit Rewards
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking
• Free Process Rewards without Process Labels
• Advancing LLM Reasoning Generalists with Preference Trees
• OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• A General Theoretical Paradigm to Understand Learning from Human Preferences
• MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Solving Quantitative Reasoning Problems with Language Models
• Self-Consistency Improves Chain of Thought Reasoning in Language Models
• Training language models to follow instructions with human feedback
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Measuring Massive Multitask Language Understanding
• Reward Tampering Problems and Solutions in Reinforcement Learning: A\n Causal Influence Diagram Perspective
• Deep reinforcement learning from human preferences
• FastCuRL: Curriculum Reinforcement Learning with Stage-wise Context Scaling for Efficient Training R1-like Reasoning Models
• OpenAI o1 System Card

Cited by

• Meta-Awareness Enhances Reasoning Models: Self-Alignment Reinforcement Learning
• V-Math: An Agentic Approach to the Vietnamese National High School Graduation Mathematics Exams

Related

• Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't
• How Difficulty-Aware Staged Reinforcement Learning Enhances LLMs' Reasoning Capabilities: A Preliminary Experimental Study
• Learning Like Humans: Advancing LLM Reasoning Capabilities via Adaptive Difficulty Curriculum Learning and Expert-Guided Self-Reformulation
• Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
• Unleashing the Reasoning Potential of Pre-trained LLMs by Critique Fine-Tuning on One Problem