SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training

2025/01/28 by Tianzhe Chu, Chu, Tianzhe, Yuexiang Zhai +15 · 7 voices · 374 citations
Computer Science · Psychology · #Domain Adaptation and Few-Shot Learning #Foundation (evidence) #Law #Mathematics education #Multimodal Machine Learning Applications #Physics #Political science #Psychology #Topic Modeling #Training (meteorology)

paper · pdf · doi:10.48550/arxiv.2501.17161

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/01/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Supervised fine-tuning (SFT) and reinforcement learning (RL) are widely used post-training techniques for foundation models. However, their roles in enhancing model generalization capabilities remain unclear. This paper studies the difference between SFT and RL on generalization and memorization, focusing on text-based rule variants and visual variants. We introduce GeneralPoints, an arithmetic reasoning card game, and adopt V-IRL, a real-world navigation environment, to assess how models trained with SFT and RL generalize to unseen variants in both textual and visual domains. We show that RL, especially when trained with an outcome-based reward, generalizes across both rule-based textual and visual variants. SFT, in contrast, tends to memorize training data and struggles to generalize out-of-distribution scenarios. Further analysis reveals that RL improves the model's underlying visual recognition capabilities, contributing to its enhanced generalization in the visual domain. Despite RL's superior generalization, we show that SFT remains essential for effective RL training; SFT stabilizes the model's output format, enabling subsequent RL to achieve its performance gains. These findings demonstrates the capability of RL for acquiring generalizable knowledge in complex, multi-modal tasks.

Citations

• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• AlphaMath Almost Zero: Process Supervision without Process
• Physics of Language Models: Part 3.1, Knowledge Storage and Extraction
• Tree of Thoughts: Deliberate Problem Solving with Large Language Models
• OPT: Open Pre-trained Transformer Language Models
• Training Compute-Optimal Large Language Models
• Training language models to follow instructions with human feedback
• On the Opportunities and Risks of Foundation Models
• Understanding deep learning (still) requires rethinking generalization
• Language Models are Few-Shot Learners

Cited by

• How Many Bits Can an Adapter Write? Measuring the Capacity and Memorization of Parameter-Efficient Fine-Tuning
• SuperFlow: Training Flow Matching Models with RL on the Fly
• The Price of Reasoning: Cost-Quality Tradeoffs in Reinforcement Learning for Neural Machine Translation
• Reasoning Before Translation: Enhancing Legal Machine Translation with Structured Reasoning
• Trace-Based On-Policy Distillation for Masked Diffusion Language Models
• SODA: Semi On-Policy Black-Box Distillation for Large Language Models
• Large-Scale Terminal Agentic Trajectory Generation from Dockerized Environments
• More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe
• Non-vacuous Generalization Bounds for Reinforcement Learning with Verifiable Rewards
• Stop Thinking, Start Looking: Efficient Post-Training for Multimodal Document Question Answering via Reasoning-Free Alignment
• Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL
• Embarrassingly Simple Self-Distillation Improves Code Generation
• Position: Modular Memory is the Key to Continual Learning Agents
• Self-Distillation Enables Continual Learning
• Learning to Orchestrate Agents in Natural Language with the Conductor
• Provable Scaling Laws of Feature Emergence from Learning Dynamics of Grokking
• The Homogenizing Effect of Large Language Models on Human Expression and Thought
• Reinforcement Learning Teachers of Test Time Scaling
• Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning
• Using Reinforcement Learning to Train Large Language Models to Explain Human Decisions
• Agentic Entropy-Balanced Policy Optimization
• Same or Not? Enhancing Visual Perception in Vision-Language Models
• A Brain-like Synergistic Core in LLMs Drives Behaviour and Learning
• Hybrid Advantage Estimation with Unified Critic for VLM Agentic Reinforcement Learning
• Enhancing Pathological VLMs with Cross-scale Reasoning
• Generalization of RLVR Using Causal Reasoning as a Testbed
• Anatomy-R1: Enhancing Anatomy Reasoning in Multimodal Large Language Models via Anatomical Similarity Curriculum and Group Diversity Augmentation
• Bridging Semantics and Geometry: A Decoupled LVLM-SAM Framework for Reasoning Segmentation in Optical Remote Sensing
• UCoder: Unsupervised Code Generation by Internal Probing of Large Language Models
• CheXPO-v2: Preference Optimization for Chest X-ray VLMs with Knowledge Graph Consistency
• Trust-Region Adaptive Policy Optimization
• AdaSearch: Balancing Parametric Knowledge and Search in Large Language Models via Reinforcement Learning
• From Facts to Conclusions : Integrating Deductive Reasoning in Retrieval-Augmented LLMs
• Beyond Fast and Slow: Cognitive-Inspired Elastic Reasoning for Large Language Models
• PuzzleCraft: Exploration-Aware Curriculum Learning for Puzzle-Based RLVR in VLMs
• Ladder Up, Memory Down: Low-Cost Fine-Tuning With Side Nets
• ADHint: Adaptive Hints with Difficulty Priors for Reinforcement Learning
• Reassessing the Role of Supervised Fine-Tuning: An Empirical Study in VLM Reasoning
• Boosting RL-Based Visual Reasoning with Selective Adversarial Entropy Intervention
• When Tables Leak: Attacking String Memorization in LLM-Based Tabular Data Generation
• Single-Agent Scaling Fails Multi-Agent Intelligence: Towards Foundation Models with Native Multi-Agent Intelligence
• Training Language Models to Use Prolog as a Tool
• Beyond Token-level Supervision: Unlocking the Potential of Decoding-based Regression via Reinforcement Learning
• Training and Evaluation of Guideline-Based Medical Reasoning in LLMs
• Context-Aware Hierarchical Learning: A Two-Step Paradigm towards Safer LLMs
• PretrainZero: Reinforcement Active Pretraining
• SPARK: Stepwise Process-Aware Rewards for Reference-Free Reinforcement Learning
• GeoZero: Incentivizing Reasoning from Scratch on Geospatial Scenes
• MRD: Multi-resolution Retrieval-Detection Fusion for High-Resolution Image Understanding
• GUI Exploration Lab: Enhancing Screen Navigation in Agents via Multi-Turn Reinforcement Learning
• Atomic Skills are the Prerequisite: When Reinforcement Learning Synthesizes Compositional Reasoning, and When It Only Amplifies
• Beyond SFT: Reinforcement Learning for Safer Large Reasoning Models with Better Reasoning Ability
• Optimizing Generative Ranking Relevance via Reinforcement Learning in Xiaohongshu Search
• From Pixels to Feelings: Aligning MLLMs with Human Cognitive Perception of Images
• Escaping the Verifier: Learning to Reason via Demonstrations
• Large Language Models' Complicit Responses to Illicit Instructions across Socio-Legal Contexts
• Thinking in 360°: Humanoid Visual Search in the Wild
• CodeV: Code with Images for Faithful Visual Reasoning via Tool-Aware Policy Optimization
• RAVEN++: Pinpointing Fine-Grained Violations in Advertisement Videos with Active Reinforcement Reasoning
• MedVision: Benchmarking Quantitative Medical Image Analysis
• Transformers with RL or SFT Provably Learn Sparse Boolean Functions, But Differently
• R2PS: Worst-Case Robust Real-Time Pursuit Strategies under Partial Observability
• TeamPath: Building MultiModal Pathology Experts with Reasoning AI Copilots
• SDA: Steering-Driven Distribution Alignment for Open LLMs without Fine-Tuning
• Let the Model Distribute Its Doubt: Confidence Estimation through Verbalized Probability Distribution
• ViSS-R1: Self-Supervised Reinforcement Video Reasoning
• Group-Aware Reinforcement Learning for Output Diversity in Large Language Models
• Tailored Primitive Initialization is the Secret Key to Reinforcement Learning
• MoralReason: Generalizable Moral Decision Alignment For LLM Agents Using Reasoning-Level Reinforcement Learning
• Look as You Think: Unifying Reasoning and Visual Evidence Attribution for Verifiable Document RAG via Reinforcement Learning
• Black-Box On-Policy Distillation of Large Language Models
• Beyond Task-Oriented and Chitchat Dialogues: Proactive and Transition-Aware Conversational Agents
• The Path Not Taken: RLVR Provably Learns Off the Principals
• From Exploration to Exploitation: A Two-Stage Entropy RLVR Approach for Noise-Tolerant MLLM Training
• SpeechJudge: Towards Human-Level Judgment for Speech Naturalness
• DigiData: Training and Evaluating General-Purpose Mobile Control Agents
• Reinforcement Learning Improves Traversal of Parametric Knowledge in LLMs
• Equilibrium Policy Generalization: A Reinforcement Learning Framework for Cross-Graph Zero-Shot Generalization in Pursuit-Evasion Games
• Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised Reinforcement Learning
• Reasoning Path Divergence: A New Metric and Curation Strategy to Unlock LLM Diverse Thinking
• PORTool: Tool-Use LLM Training with Rewarded Tree
• Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoning
• InstructPLM-mu: 1-Hour Fine-Tuning of ESM2 Beats ESM3 in Protein Mutation Predictions
• VerIF: Verification Engineering for Reinforcement Learning in Instruction Following
• Urban-R1: Reinforced MLLMs Mitigate Geospatial Biases for Urban General Intelligence
• RL makes MLLMs see better than SFT
• Metis-SPECS: Decoupling Multimodal Learning via Self-distilled Preference-based Cold Start
• A Survey on Unlearning in Large Language Models
• Think before Recommendation: Autonomous Reasoning-enhanced Recommender
• CityRiSE: Reasoning Urban Socio-Economic Status in Vision-Language Models via Reinforcement Learning
• Generalization or Memorization: Dynamic Decoding for Mode Steering
• Boosting Accuracy and Efficiency of Budget Forcing in LLMs via Reinforcement Learning for Mathematical Reasoning
• Enhancing visual-LLM for construction site safety compliance via prompt engineering and Bi-stage retrieval-augmented generation
• Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting
• Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsing
• InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Training
• Composition-Grounded Data Synthesis for Visual Reasoning
• RoboGPT-R1: Enhancing Robot Planning with Reinforcement Learning
• SimKO: Simple Pass@K Policy Optimization
• ChatR1: Reinforcement Learning for Conversational Reasoning and Retrieval Augmented Question Answering
• QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMs
• Reasoning as Representation: Rethinking Visual Reinforcement Learning in Image Quality Assessment
• Parallel Scaling Law: Unveiling Reasoning Generalization through A Cross-Linguistic Perspective
• Chart-RVR: Reinforcement Learning with Verifiable Rewards for Explainable Chart Reasoning
• More Than One Teacher: Adaptive Multi-Guidance Policy Optimization for Diverse Exploration
• Cog-Rethinker: Hierarchical Metacognitive Reinforcement Learning for LLM Reasoning
• Enhancing Large Language Model Reasoning via Selective Critical Token Fine-Tuning
• OpusAnimation: Code-Based Dynamic Chart Generation
• Don't Just Fine-tune the Agent, Tune the Environment
• Synthetic Sandbox for Training Machine Learning Engineering Agents
• Can RL Improve Generalization of LLM Agents? An Empirical Study
• Autonomous Agents for Scientific Discovery: Orchestrating Scientists, Language, Code, and Physics
• HINT: Helping Ineffective Rollouts Navigate Towards Effectiveness
• DSPO: Stable and Efficient Policy Optimization for Agentic Search and Reasoning
• Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models
• Agent Learning via Early Experience
• TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevance
• Empirical Comparison of Membership Inference Attacks in Deep Transfer Learning
• Reinforcement Learning for Clinical Reasoning: Aligning LLMs with ACR Imaging Appropriateness Criteria
• Mitigating Forgetting Between Supervised and Reinforcement Learning Yields Stronger Reasoners
• Making Mathematical Reasoning Adaptive
• RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learning
• Attack via Overfitting: 10-shot Benign Fine-tuning to Jailbreak LLMs
• Retrv-R1: A Reasoning-Driven MLLM Framework for Universal and Efficient Multimodal Retrieval
• Graph-S3: Enhancing Agentic textual Graph Retrieval with Synthetic Stepwise Supervision
• Beyond Log Likelihood: Probability-Based Objectives for Supervised Fine-Tuning across the Model Capability Continuum
• Debunk the Myth of SFT Generalization
• One-Token Rollout: Guiding Supervised Fine-Tuning of LLMs with Policy Gradient
• R-Log: Incentivizing Log Analysis Capability in LLMs via Reasoning-based Reinforcement Learning
• More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models
• TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning
• Thinking Sparks!: Emergent Attention Heads in Reasoning Models During Post Training
• UniAPL: A Unified Adversarial Preference Learning Framework for Instruct-Following
• Scaling Generalist Data-Analytic Agents
• SemanticShield: LLM-Powered Audits Expose Shilling Attacks in Recommender Systems
• When Greedy Wins: Emergent Exploitation Bias in Meta-Bandit LLM Training
• Pushing LLMs to Their Logical Reasoning Bound: The Role of Data Reasoning Intensity
• FrameThinker: Learning to Think with Long Videos via Multi-Turn Frame Spotlighting
• Reasoning or Retrieval? A Study of Answer Attribution on Large Reasoning Models
• PIPer: On-Device Environment Setup via Online Reinforcement Learning
• Rethinking Reward Miscalibration of GRPO in Agentic RL
• Anchored Supervised Fine-Tuning
• Beyond English-Centric Training: How Reinforcement Learning Improves Cross-Lingual Reasoning in LLMs
• PaperSearchQA: Learning to Search and Reason over Scientific Papers with RLVR
• Your Models Have Thought Enough: Training Large Reasoning Models to Stop Overthinking
• Dynamic-TreeRPO: Breaking the Independent Trajectory Bottleneck with Structured Sampling
• Test-Time Policy Adaptation for Enhanced Multi-Turn Interactions with LLMs
• VideoScore2: Think before You Score in Generative Video Evaluation
• CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning
• Variational Reasoning for Language Models
• IA2: Alignment with ICL Activations Improves Supervised Fine-Tuning
• Benefits and Pitfalls of Reinforcement Learning for Language Model Planning: A Theoretical Perspective
• Rule-Based Reinforcement Learning for Document Image Classification with Vision Language Models
• The Rogue Scalpel: Activation Steering Compromises LLM Safety
• Unlocking the Essence of Beauty: Advanced Aesthetic Reasoning with Relative-Absolute Policy Optimization
• ResT: Reshaping Token-Level Policy Gradients for Tool-Use Large Language Models
• Tree Search for LLM Agent Reinforcement Learning
• Reinforcement Learning Fine-Tuning Enhances Activation Intensity and Diversity in the Internal Circuitry of LLMs
• SFT Doesn't Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMs
• RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs
• DRES: Benchmarking LLMs for Disfluency Removal
• ThinkFake: Reasoning in Multimodal Large Language Models for AI-Generated Image Detection
• VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models
• EnAnchored-X2X: English-Anchored Optimization for Many-to-Many Translation
• UserRL: Training Interactive User-Centric Agent via Reinforcement Learning
• bi-GRPO: Bidirectional Optimization for Jailbreak Backdoor Injection on LLMs
• From Scoring to Acting: Outcome-Verified Comparative Self-Distillation for LLM Agents
• FinSMART: Financial Sentiment Analysis for Algorithmic Trading through Market-Aligned Reinforcement Learning
• Would You Walk to the Car Wash? Revealing the Salience Bias of Large Language Models in Commonsense Reasoning
• S-GRPO: Unified Post-Training for Large Vision-Language Models
• Proximal Supervised Fine-Tuning
• Reinforcement Learning on Pre-Training Data
• MAPO: Mixed Advantage Policy Optimization
• Correlation or Causation: Analyzing the Causal Structures of LLM and LRM Reasoning Process
• Generalizable End-to-End Tool-Use RL with Synthetic CodeGym
• Understanding Post-Training Structural Changes in Large Language Models
• Can GRPO Boost Complex Multimodal Table Understanding?
• Enhancing Generative Auto-bidding with Offline Reward Evaluation and Policy Search
• Compute as Teacher: Turning Inference Compute Into Reference-Free Supervision
• Explain Before You Answer: A Survey on Compositional Visual Reasoning
• Building Coding Agents via Entropy-Enhanced Multi-Turn Preference Optimization
• LaV-CoT: Language-Aware Visual CoT with Multi-Aspect Reward Optimization for Real-World Multilingual VQA
• PaVeRL-SQL: Text-to-SQL via Partial-Match Rewards and Verbal Reinforcement Learning
• Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoning
• Staying in the Sweet Spot: Responsive Reasoning Evolution via Capability-Adaptive Hint Scaffolding
• RL's Razor: Why Online Reinforcement Learning Forgets Less
• A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models
• Learning to Generate Unit Test via Adversarial Reinforcement Learning
• MedGR²: Breaking the Data Barrier for Medical Reasoning via Generative Reward Learning
• Token Buncher: Shielding LLMs from Harmful Reinforcement Learning Fine-Tuning
• AR²: Adversarial Reinforcement Learning for Abstract Reasoning in Large Language Models
• SWIRL: A Staged Workflow for Interleaved Reinforcement Learning in Mobile GUI Control
• VERIRL: Boosting the LLM-based Verilog Code Generation via Reinforcement Learning
• Toward Better EHR Reasoning in LLMs: Reinforcement Learning with Expert Attention Guidance
• ALIGN: Word Association Learning for Cultural Alignment in Large Language Models
• Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulation
• G²RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidance
• Atom-Searcher: Enhancing Agentic Deep Research via Fine-Grained Atomic Thought Reward
• On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting
• UAV-VL-R1: Generalizing Vision-Language Models via Supervised Fine-Tuning and Multi-Stage GRPO for UAV Visual Reasoning
• Diversity First, Quality Later: A Two-Stage Assumption for Language Model Alignment
• Pruning Long Chain-of-Thought of Large Reasoning Models via Small-Scale Preference Optimization
• STELAR-VISION: Self-Topology-Aware Efficient Learning for Aligned Reasoning in Vision
• DocThinker: Explainable Multimodal Large Language Models with Rule-based Reinforcement Learning for Document Understanding
• Learning to Align, Aligning to Learn: A Unified Approach for Self-Optimized Alignment
• From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
• EvoCoT: Overcoming the Exploration Bottleneck in Reinforcement Learning
• Sample-efficient LLM Optimization with Reset Replay
• On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification
• Shuffle-R1: Efficient RL framework for Multimodal Large Language Models via Data-centric Dynamic Shuffle
• Decoupling Understanding from Reasoning via Problem Space Mapping for Small-Scale Model Reasoning
• GuirlVG: Incentivize GUI Visual Grounding via Empirical Exploration on Reinforcement Learning
• AttriLens-Mol: Attribute Guided Reinforcement Learning for Molecular Property Prediction with Large Language Models
• VRPRM: Process Reward Modeling via Visual Reasoning
• Beyond Surface-Level Detection: Towards Cognitive-Driven Defense Against Jailbreak Attacks via Meta-Operations Reasoning
• Tool-integrated Reinforcement Learning for Repo Deep Search
• A Rolling Stone Gathers No Moss: Adaptive Policy Optimization for Stable Self-Evaluation in Large Multimodal Models
• Reconsidering Overthinking: Penalizing Internal and External Redundancy in CoT Reasoning
• MolReasoner: Toward Effective and Interpretable Reasoning for Molecular LLMs
• SA-GCS: Semantic-Aware Gaussian Curriculum Scheduling for UAV Vision-Language Navigation
• PilotRL: Training Language Model Agents via Global Planning-Guided Progressive Reinforcement Learning
• Reasoning Language Models for Root Cause Analysis in 5G Wireless Networks
• Flow Matching Policy Gradients
• Learning Only with Images: Visual Reinforcement Learning with Reasoning, Rendering, and Visual Feedback
• IM-Chat: A Multi-agent LLM Framework Integrating Tool-Calling and Diffusion Modeling for Knowledge Transfer in Injection Molding Industry
• PurpCode: Reasoning for Safer Code Generation
• FinDPO: Financial Sentiment Analysis for Algorithmic Trading through Preference Optimization of LLMs
• MeMo: Memory as a Model
• BusterX++: Towards Unified Cross-Modal AI-Generated Content Detection and Explanation with MLLM
• Can One Domain Help Others? A Data-Centric Study on Multi-Domain Reasoning via Reinforcement Learning
• Reinforcement Learning Fine-Tunes a Sparse Subnetwork in Large Language Models
• Semi-off-Policy Reinforcement Learning for Vision-Language Slow-Thinking Reasoning
• Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities
• LaViPlan : Language-Guided Visual Path Planning with RLVR
• Xiangqi-R1: Enhancing Spatial Strategic Reasoning in LLMs for Chinese Chess via Reinforcement Learning
• RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic Sampling
• SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning
• RedOne: Revealing Domain-specific LLM Post-Training in Social Networking Services
• Reinforce LLM Reasoning through Multi-Agent Reflection
• Teaching LLM to Reason: Reinforcement Learning from Algorithmic Problems without Code
• Data Compressibility Quantifies LLM Memorization
• Knowing When to Quit: Diagnosing and Training LLMs to Abort Futile Reasoning
• Open Vision Reasoner: Transferring Linguistic Cognitive Behavior for Visual Reasoning
• Replacing thinking with tool usage enables reasoning in small language models
• R1-RE: Cross-Domain Relation Extraction with RLVR
• Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Training
• Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?
• ESSA: Evolutionary Strategies for Scalable Alignment
• Enhancing Spatial Reasoning in Vision-Language Models via Chain-of-Thought Prompting and Reinforcement Learning
• StepHint: Multi-level Stepwise Hints Enhance Reinforcement Learning to Reason
• WebSailor: Navigating Super-human Reasoning for Web Agent
• Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMs
• Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling
• Reasoner for Real-World Event Detection: Scaling Reinforcement Learning via Adaptive Perplexity-Aware Sampling Strategy
• Improving the Reasoning of Multi-Image Grounding in MLLMs via Reinforcement Learning
• GeoRA: Geometry-Aware Low-Rank Adaptation for RLVR
• TGRPO :Fine-tuning Vision-Language-Action Model via Trajectory-wise Group Relative Policy Optimization
• User Behavior Prediction as a Generic, Robust, Scalable, and Low-Cost Evaluation Strategy for Estimating Generalization in LLMs
• RAG-R1: Incentivizing the Search and Reasoning Capabilities of LLMs through Multi-query Parallelism
• Empowering Small VLMs to Think with Dynamic Memorization and Exploration
• Listener-Rewarded Thinking in VLMs for Image Preferences
• Robust and Efficient Autoregressive Speech Synthesis with Dynamic Chunk-wise Prediction Policy
• Exploring Task-Solving Paradigm for Generalized Cross-Domain Face Anti-Spoofing via Reinforcement Fine-Tuning
• RSafe: Incentivizing proactive reasoning to build robust and adaptive LLM safeguards
• Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions
• ReCode: Updating Code API Knowledge with Reinforcement Learning
• SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning
• Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning
• RePIC: Reinforced Post-Training for Personalizing Multi-Modal Language Models
• Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning
• WeThink: Toward General-purpose Vision-Language Reasoning via Reinforcement Learning
• GEMeX-RMCoT: An Enhanced Med-VQA Dataset for Region-Aware Multimodal Chain-of-Thought Reasoning
• BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning
• Chiron-o1: Igniting Multimodal Large Language Models towards Generalizable Medical Reasoning via Mentor-Intern Collaborative Search
• ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving
• Play to Generalize: Learning to Reason Through Game Play
• Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Models
• Continual Learning for Generative AI: From LLMs to MLLMs and Beyond
• Offline RL by Reward-Weighted Fine-Tuning for Conversation Optimization
• Learning a Continue-Thinking Token for Enhanced Test-Time Scaling
• Time Series Forecasting as Reasoning: A Slow-Thinking Approach with Reinforced LLMs
• LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learning
• Writing-RL: Advancing Long-form Writing via Adaptive Curriculum Reinforcement Learning
• Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning
• Rex-Thinker: Grounded Object Referring via Chain-of-Thought Reasoning
• KDRL: Post-Training Reasoning LLMs via Unified Knowledge Distillation and Reinforcement Learning
• Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning
• Knowledge or Reasoning? A Close Look at How LLMs Think Across Domains
• Reasoning-Table: Exploring Reinforcement Learning for Table Reasoning
• Generalizable LLM Learning of Graph Synthetic Data with Post-training Alignment
• LIFT the Veil for the Truth: Principal Weights Emerge after Rank Reduction for Reasoning-Focused Supervised Fine-Tuning
• Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsing
• Reasoning Like an Economist: Post-Training on Economic Problems Induces Strategic Generalization in LLMs
• TimeHC-RL: Temporal-aware Hierarchical Cognitive Reinforcement Learning for Enhancing LLMs' Social Intelligence
• RAST: Reasoning Activation in LLMs via Small-model Transfer
• ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models
• InfiMed: Low-Resource Medical MLLMs with Advancing Understanding and Reasoning
• Self-Correcting Code Generation Using Small Language Models
• DIP-R1: Deep Inspection and Perception with RL Looking Through and Understanding Complex Scenes
• Learning to Route Queries Across Knowledge Bases for Step-wise Retrieval-Augmented Reasoning
• Spatial Knowledge Graph-Guided Multimodal Synthesis
• Beyond path selection: Better LLMs for Scientific Information Extraction with MimicSFT and Relevance and Rule-induced(R²)GRPO
• SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learning
• TWIST: Teleoperated Whole-Body Imitation System
• RM-R1: Reward Modeling as Reasoning
• What Can RL Bring to VLA Generalization? An Empirical Study
• R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learning
• Deciphering Trajectory-Aided LLM Reasoning: An Optimization Perspective
• Unveiling the Compositional Ability Gap in Vision-Language Reasoning Model
• Collision- and Reachability-Aware Multi-Robot Control with Grounded LLM Planners
• G1: Teaching LLMs to Reason on Graphs with Reinforcement Learning
• Bridging Supervised Learning and Reinforcement Learning in Math Reasoning
• Towards Revealing the Effectiveness of Small-Scale Fine-tuning in R1-style Reinforcement Learning
• Beyond Distillation: Pushing the Limits of Medical LLM Reasoning with Minimalist Rule-Based RL
• R1-Searcher++: Incentivizing the Dynamic Knowledge Acquisition of LLMs via Reinforcement Learning
• RLKD: Distilling LLMs' Reasoning via Reinforcement Learning
• UFT: Unifying Supervised and Reinforcement Fine-Tuning
• SATURN: SAT-based Reinforcement Learning to Unleash LLMs Reasoning
• SafeKey: Amplifying Aha-Moment Insights for Safety Reasoning
• Procedural Environment Generation for Tool-Use Agents
• AvatarShield: Visual Reinforcement Learning for Human-Centric Synthetic Video Detection
• NOVER: Incentive Training for Language Models via Verifier-Free Reinforcement Learning
• Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignment
• StepSearch: Igniting LLMs Search Ability via Step-Wise Proximal Policy Optimization
• RL Tango: Reinforcing Generator and Verifier Together for Language Reasoning
• ConvSearch-R1: Enhancing Query Reformulation for Conversational Search with Reasoning via Reinforcement Learning
• RoT: Enhancing Table Reasoning with Iterative Row-Wise Traversals
• Scaling Reasoning, Losing Control: Evaluating Instruction Following in Large Reasoning Models
• Visionary-R1: Mitigating Shortcuts in Visual Reasoning with Reinforcement Learning
• Process vs. Outcome Reward: Which is Better for Agentic RAG Reinforcement Learning
• Unlocking the Potential of Difficulty Prior in RL-based Multimodal Reasoning
• Observe-R1: Unlocking Reasoning Abilities of MLLMs with Dynamic Progressive Reinforcement Learning
• Visual Planning: Let's Think Only with Images
• Critique-Guided Distillation for Robust Reasoning via Refinement
• Disentangling Reasoning and Knowledge in Medical Large Language Models
• Search and Refine During Think: Facilitating Knowledge Refinement for Improved Retrieval-Augmented Reasoning
• OpenThinkIMG: Learning to Think with Images via Visual Tool Reinforcement Learning
• SLMs as Multi-Agent Routers: A Progressive SFT and Reinforcement Learning Approach
• Bridging AI and Carbon Capture: A Dataset for LLMs in Ionic Liquids and CBE Research
• VLA-OPD: Bridging Offline SFT and Online RL for Vision-Language-Action Models via On-Policy Distillation
• Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models
• RL Post-Training Builds Compositional Reasoning Strategies
• Can Agents Generalize to the Open World? Unveiling the Fragility of Static Training in Tool Use
• 100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Models
• SFT Overtraining Predicts Rank Inversion via Entropy Collapse Under RLVR
• From Reasoning Traces to Reusable Modules: Understanding Compositional Generalization in Language Model Reasoning
• SmallPlan: Leverage Small Language Models for Sequential Path Planning with Simulation-Powered, LLM-Guided Distillation
• Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Models
• On Training Large Language Models for Long-Horizon Tasks: An Empirical Study of Horizon Length
• SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoning
• Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Models
• SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning
• Minerva: Reinforcement Learning with Verifiable Rewards for Cyber Threat Intelligence LLMs
• Reason Like a Radiologist: Chain-of-Thought and Reinforcement Learning for Verifiable Report Generation
• An Imperfect Verifier is Good Enough: Learning with Noisy Rewards
• SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs
• MOON3.0: Reasoning-aware Multimodal Representation Learning for E-commerce Product Understanding
• TTRL: Test-Time Reinforcement Learning
• Reasoning Physical Video Generation with Diffusion Timestep Tokens via Reinforcement Learning
• Seeing from Another Perspective: Evaluating Multi-View Understanding in MLLMs
• SWE-Synth: Synthesizing Verifiable Bug-Fix Data to Enable Large Language Models in Resolving Real-World Bugs
• Relation-R1: Progressively Cognitive Chain-of-Thought Guided Reinforcement Learning for Unified Relation Comprehension
• ToolRL: Reward is All Tool Learning Needs
• d1: Scaling Reasoning in Diffusion Large Language Models via Reinforcement Learning
• Nemotron-CrossThink: Scaling Self-Learning beyond Math Reasoning
• Measuring LLM Novelty As The Frontier Of Original And High-Quality Output
• Slow Thinking for Sequential Recommendation
• Playpen: An Environment for Exploring Learning Through Conversational Interaction
• Perception-R1: Pioneering Perception Policy with Reinforcement Learning
• Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining
• VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model
• SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models
• Thinking Out Loud: Do Reasoning Models Know When They're Right?
• Rethinking RL Scaling for Vision Language Models: A Transparent, From-Scratch Framework and Comprehensive Evaluation Scheme

Discussions

• arxiv.org/abs/2501.17161 Yes! [bsky, 27 points, 2 comments]
• 8/ Paper: "SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training" 📝 arxiv.org/abs/2501.17161 [bsky, 5 points, 0 comments]
• Supervised Fine-Tuning Memorizes, RL Generalizes [hn, 1 points, 0 comments]
• SFT Memorizes,RL Generalizes: Comparative Study of Foundation Model PostTraining [hn, 1 points, 0 comments]
• Paper: arxiv.org/abs/2501.171... Project page: tianzhechu.com/SFTvsRL/ [bsky, 0 points, 0 comments]
• arxiv.org/abs/2501.17161 [bsky, 0 points, 1 comments]
• [Weekend Read] SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training - arxiv.org/pdf/2501.17161 - Using Reinforcement Learning (RL) help generalizing and SFT help stabil [bsky, 0 points, 0 comments]

Related

• Debunk the Myth of SFT Generalization
• Why Does Reinforcement Learning Generalize? A Feature-Level Mechanistic Study of Post-Training in Large Language Models
• Reassessing the Role of Supervised Fine-Tuning: An Empirical Study in VLM Reasoning
• Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead
• SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models