Atomic Skills are the Prerequisite: When Reinforcement Learning Synthesizes Compositional Reasoning, and When It Only Amplifies

2025/12/01 by Sitao Cheng, Cheng, Sitao, Yin, Xunjian +12 · 1 citation
Computer Science · Psychology · #Artificial Intelligence (cs.AI) #Child and Animal Learning Development #Computation and Language (cs.CL) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Reinforcement Learning in Robotics

paper · pdf · doi:10.48550/arxiv.2512.01970

openalex publication_date 2025/12/01 · openalex created_date 2025/12/03 · openalex updated_date 2026/08/01

Abstract

Does Reinforcement Learning (RL) merely amplify existing skills, or synthesize novel skills? We investigate this question through the lens of Complementary Reasoning: the critical practical capability of integrating internal knowledge with external context, a prerequisite for reliable Continual Learning and Retrieval-Augmented Generation. To avoid pre-training contamination, we construct a controlled semanticsynthetic dataset of biographies and decompose this capability into two atomic skills: Parametric Reasoning (retrieving facts encoded in model weights) and Contextual Reasoning (processing novel in-context information). We present two findings. First, models supervised directly on the composite task reach high accuracy on seen facts and reasoning paths (90%) but collapse on novel facts and reasoning paths (18%), indicating that Supervised Fine-Tuning (SFT) relies on rote memorization rather than genuine skill integration. Second, RL bridges this generalization gap, acting as a skill synthesizer rather than a mere amplifier--but only under a strict prerequisite: it synthesizes new composite strategies only when the base model has first mastered the independent atomic skills via SFT. These results suggest that decoupled atomic training followed by RL offers a scalable path to complex novel reasoning.

Citations

• From f(x) and g(x) to f(g(x)): LLMs Learn New Skills in RL by Composing Old Ones
• e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMs
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
• SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training
• Disentangling Memory and Reasoning Ability in Large Language Models
• Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data
• Thread: A Logic-Based Data Organization Paradigm for How-To Question Answering with Retrieval Augmented Generation
• Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization
• Call Me When Necessary: LLMs can Efficiently and Faithfully Reason over Structured Environments
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• MarkQA: A large scale KBQA dataset with numerical reasoning
• ALCUNA: Large Language Models Meet New Knowledge
• Physics of Language Models: Part 3.2, Knowledge Manipulation
• Physics of Language Models: Part 3.1, Knowledge Storage and Extraction
• Question Decomposition Tree for Answering Complex Questions over Knowledge Bases
• Mind2Web: Towards a Generalist Agent for the Web
• GPT-4 Technical Report
• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
• Proximal Policy Optimization Algorithms
• The Invisible Leash: Why RLVR May or May Not Escape Its Origin

Cited by

• Differentiable Evolutionary Reinforcement Learning

Related

• Laying the Foundation First? Investigating the Generalization from Atomic Skills to Complex Reasoning Tasks
• On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models
• RL for Reasoning by Adaptively Revealing Rationales
• SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training
• From Reasoning Traces to Reusable Modules: Understanding Compositional Generalization in Language Model Reasoning