SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training

2025/01/28 by Tianzhe Chu, Chu, Tianzhe, Yuexiang Zhai +15 · 7 voices · 220 citations
Computer Science · #Multimodal Machine Learning Applications #Topic Modeling #Domain Adaptation and Few-Shot Learning

paper · pdf · doi:10.48550/arxiv.2501.17161

Abstract

Supervised fine-tuning (SFT) and reinforcement learning (RL) are widely used post-training techniques for foundation models. However, their roles in enhancing model generalization capabilities remain unclear. This paper studies the difference between SFT and RL on generalization and memorization, focusing on text-based rule variants and visual variants. We introduce GeneralPoints, an arithmetic reasoning card game, and adopt V-IRL, a real-world navigation environment, to assess how models trained with SFT and RL generalize to unseen variants in both textual and visual domains. We show that RL, especially when trained with an outcome-based reward, generalizes across both rule-based textual and visual variants. SFT, in contrast, tends to memorize training data and struggles to generalize out-of-distribution scenarios. Further analysis reveals that RL improves the model's underlying visual recognition capabilities, contributing to its enhanced generalization in the visual domain. Despite RL's superior generalization, we show that SFT remains essential for effective RL training; SFT stabilizes the model's output format, enabling subsequent RL to achieve its performance gains. These findings demonstrates the capability of RL for acquiring generalizable knowledge in complex, multi-modal tasks.

Citations

• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• AlphaMath Almost Zero: Process Supervision without Process
• Physics of Language Models: Part 3.1, Knowledge Storage and Extraction
• Tree of Thoughts: Deliberate Problem Solving with Large Language Models
• OPT: Open Pre-trained Transformer Language Models
• Training Compute-Optimal Large Language Models
• Training language models to follow instructions with human feedback
• On the Opportunities and Risks of Foundation Models
• Understanding deep learning (still) requires rethinking generalization
• Language Models are Few-Shot Learners

Cited by

• How Many Bits Can an Adapter Write? Measuring the Capacity and Memorization of Parameter-Efficient Fine-Tuning
• SuperFlow: Training Flow Matching Models with RL on the Fly
• The Price of Reasoning: Cost-Quality Tradeoffs in Reinforcement Learning for Neural Machine Translation
• Reasoning Before Translation: Enhancing Legal Machine Translation with Structured Reasoning
• Trace-Based On-Policy Distillation for Masked Diffusion Language Models
• SODA: Semi On-Policy Black-Box Distillation for Large Language Models
• Large-Scale Terminal Agentic Trajectory Generation from Dockerized Environments
• More with Less: a Large Scale Remote Sensing VLM with a Simple Recipe
• Non-vacuous Generalization Bounds for Reinforcement Learning with Verifiable Rewards
• Stop Thinking, Start Looking: Efficient Post-Training for Multimodal Document Question Answering via Reasoning-Free Alignment
• Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL
• Embarrassingly Simple Self-Distillation Improves Code Generation
• Position: Modular Memory is the Key to Continual Learning Agents
• Self-Distillation Enables Continual Learning
• Learning to Orchestrate Agents in Natural Language with the Conductor
• Provable Scaling Laws of Feature Emergence from Learning Dynamics of Grokking
• The Homogenizing Effect of Large Language Models on Human Expression and Thought
• Reinforcement Learning Teachers of Test Time Scaling
• Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoning
• Using Reinforcement Learning to Train Large Language Models to Explain Human Decisions
• Agentic Entropy-Balanced Policy Optimization
• Same or Not? Enhancing Visual Perception in Vision-Language Models
• A Brain-like Synergistic Core in LLMs Drives Behaviour and Learning
• Hybrid Advantage Estimation with Unified Critic for VLM Agentic Reinforcement Learning
• Enhancing Pathological VLMs with Cross-scale Reasoning
• Generalization of RLVR Using Causal Reasoning as a Testbed
• Anatomy-R1: Enhancing Anatomy Reasoning in Multimodal Large Language Models via Anatomical Similarity Curriculum and Group Diversity Augmentation
• Bridging Semantics and Geometry: A Decoupled LVLM-SAM Framework for Reasoning Segmentation in Optical Remote Sensing
• UCoder: Unsupervised Code Generation by Internal Probing of Large Language Models
• CheXPO-v2: Preference Optimization for Chest X-ray VLMs with Knowledge Graph Consistency
• Trust-Region Adaptive Policy Optimization
• AdaSearch: Balancing Parametric Knowledge and Search in Large Language Models via Reinforcement Learning
• From Facts to Conclusions : Integrating Deductive Reasoning in Retrieval-Augmented LLMs
• Beyond Fast and Slow: Cognitive-Inspired Elastic Reasoning for Large Language Models
• PuzzleCraft: Exploration-Aware Curriculum Learning for Puzzle-Based RLVR in VLMs
• Ladder Up, Memory Down: Low-Cost Fine-Tuning With Side Nets
• ADHint: Adaptive Hints with Difficulty Priors for Reinforcement Learning
• Reassessing the Role of Supervised Fine-Tuning: An Empirical Study in VLM Reasoning
• Boosting RL-Based Visual Reasoning with Selective Adversarial Entropy Intervention
• When Tables Leak: Attacking String Memorization in LLM-Based Tabular Data Generation
• Single-Agent Scaling Fails Multi-Agent Intelligence: Towards Foundation Models with Native Multi-Agent Intelligence
• Training Language Models to Use Prolog as a Tool
• Beyond Token-level Supervision: Unlocking the Potential of Decoding-based Regression via Reinforcement Learning
• Training and Evaluation of Guideline-Based Medical Reasoning in LLMs
• Context-Aware Hierarchical Learning: A Two-Step Paradigm towards Safer LLMs
• PretrainZero: Reinforcement Active Pretraining
• SPARK: Stepwise Process-Aware Rewards for Reference-Free Reinforcement Learning
• GeoZero: Incentivizing Reasoning from Scratch on Geospatial Scenes
• MRD: Multi-resolution Retrieval-Detection Fusion for High-Resolution Image Understanding
• GUI Exploration Lab: Enhancing Screen Navigation in Agents via Multi-Turn Reinforcement Learning
• From Atomic to Composite: Reinforcement Learning Enables Generalization in Complementary Reasoning
• Beyond SFT: Reinforcement Learning for Safer Large Reasoning Models with Better Reasoning Ability
• Optimizing Generative Ranking Relevance via Reinforcement Learning in Xiaohongshu Search
• From Pixels to Feelings: Aligning MLLMs with Human Cognitive Perception of Images
• Escaping the Verifier: Learning to Reason via Demonstrations
• Large Language Models' Complicit Responses to Illicit Instructions across Socio-Legal Contexts
• Thinking in 360°: Humanoid Visual Search in the Wild
• CodeV: Code with Images for Faithful Visual Reasoning via Tool-Aware Policy Optimization
• RAVEN++: Pinpointing Fine-Grained Violations in Advertisement Videos with Active Reinforcement Reasoning
• MedVision: Benchmarking Quantitative Medical Image Analysis
• Transformers with RL or SFT Provably Learn Sparse Boolean Functions, But Differently
• R2PS: Worst-Case Robust Real-Time Pursuit Strategies under Partial Observability
• TeamPath: Building MultiModal Pathology Experts with Reasoning AI Copilots
• SDA: Steering-Driven Distribution Alignment for Open LLMs without Fine-Tuning
• Let the Model Distribute Its Doubt: Confidence Estimation through Verbalized Probability Distribution
• ViSS-R1: Self-Supervised Reinforcement Video Reasoning
• Group-Aware Reinforcement Learning for Output Diversity in Large Language Models
• Tailored Primitive Initialization is the Secret Key to Reinforcement Learning
• MoralReason: Generalizable Moral Decision Alignment For LLM Agents Using Reasoning-Level Reinforcement Learning
• Look as You Think: Unifying Reasoning and Visual Evidence Attribution for Verifiable Document RAG via Reinforcement Learning
• Black-Box On-Policy Distillation of Large Language Models
• Beyond Task-Oriented and Chitchat Dialogues: Proactive and Transition-Aware Conversational Agents
• The Path Not Taken: RLVR Provably Learns Off the Principals
• From Exploration to Exploitation: A Two-Stage Entropy RLVR Approach for Noise-Tolerant MLLM Training
• SpeechJudge: Towards Human-Level Judgment for Speech Naturalness
• DigiData: Training and Evaluating General-Purpose Mobile Control Agents
• Reinforcement Learning Improves Traversal of Parametric Knowledge in LLMs
• Equilibrium Policy Generalization: A Reinforcement Learning Framework for Cross-Graph Zero-Shot Generalization in Pursuit-Evasion Games
• Spatial-SSRL: Enhancing Spatial Understanding via Self-Supervised Reinforcement Learning
• Reasoning Path Divergence: A New Metric and Curation Strategy to Unlock LLM Diverse Thinking
• PORTool: Tool-Use LLM Training with Rewarded Tree
• Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoning
• InstructPLM-mu: 1-Hour Fine-Tuning of ESM2 Beats ESM3 in Protein Mutation Predictions
• Urban-R1: Reinforced MLLMs Mitigate Geospatial Biases for Urban General Intelligence
• RL makes MLLMs see better than SFT
• Metis-SPECS: Decoupling Multimodal Learning via Self-distilled Preference-based Cold Start
• A Survey on Unlearning in Large Language Models
• Think before Recommendation: Autonomous Reasoning-enhanced Recommender
• CityRiSE: Reasoning Urban Socio-Economic Status in Vision-Language Models via Reinforcement Learning
• Generalization or Memorization: Dynamic Decoding for Mode Steering
• Boosting Accuracy and Efficiency of Budget Forcing in LLMs via Reinforcement Learning for Mathematical Reasoning
• Enhancing visual-LLM for construction site safety compliance via prompt engineering and Bi-stage retrieval-augmented generation
• Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting
• Infinity Parser: Layout Aware Reinforcement Learning for Scanned Document Parsing
• InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Training
• Composition-Grounded Instruction Synthesis for Visual Reasoning
• RoboGPT-R1: Enhancing Robot Planning with Reinforcement Learning
• SimKO: Simple Pass@K Policy Optimization
• ChatR1: Reinforcement Learning for Conversational Reasoning and Retrieval Augmented Question Answering
• QeRL: Beyond Efficiency -- Quantization-enhanced Reinforcement Learning for LLMs
• Reasoning as Representation: Rethinking Visual Reinforcement Learning in Image Quality Assessment
• Parallel Scaling Law: Unveiling Reasoning Generalization through A Cross-Linguistic Perspective
• Chart-RVR: Reinforcement Learning with Verifiable Rewards for Explainable Chart Reasoning
• More Than One Teacher: Adaptive Multi-Guidance Policy Optimization for Diverse Exploration
• Cog-Rethinker: Hierarchical Metacognitive Reinforcement Learning for LLM Reasoning
• Enhancing Large Language Model Reasoning via Selective Critical Token Fine-Tuning
• OpusAnimation: Code-Based Dynamic Chart Generation
• Don't Just Fine-tune the Agent, Tune the Environment
• Synthetic Sandbox for Training Machine Learning Engineering Agents
• Can RL Improve Generalization of LLM Agents? An Empirical Study
• Autonomous Agents for Scientific Discovery: Orchestrating Scientists, Language, Code, and Physics
• HINT: Helping Ineffective Rollouts Navigate Towards Effectiveness
• DSPO: Stable and Efficient Policy Optimization for Agentic Search and Reasoning
• Detecting Data Contamination from Reinforcement Learning Post-training for Large Language Models
• Agent Learning via Early Experience
• TaoSR-AGRL: Adaptive Guided Reinforcement Learning Framework for E-commerce Search Relevance
• Empirical Comparison of Membership Inference Attacks in Deep Transfer Learning
• Reinforcement Learning for Clinical Reasoning: Aligning LLMs with ACR Imaging Appropriateness Criteria
• Mitigating Forgetting Between Supervised and Reinforcement Learning Yields Stronger Reasoners
• Making Mathematical Reasoning Adaptive
• RewardMap: Tackling Sparse Rewards in Fine-grained Visual Reasoning via Multi-Stage Reinforcement Learning
• Attack via Overfitting: 10-shot Benign Fine-tuning to Jailbreak LLMs
• Retrv-R1: A Reasoning-Driven MLLM Framework for Universal and Efficient Multimodal Retrieval
• Graph-S3: Enhancing Agentic textual Graph Retrieval with Synthetic Stepwise Supervision
• Beyond Log Likelihood: Probability-Based Objectives for Supervised Fine-Tuning across the Model Capability Continuum
• Debunk the Myth of SFT Generalization
• One-Token Rollout: Guiding Supervised Fine-Tuning of LLMs with Policy Gradient
• R-Log: Incentivizing Log Analysis Capability in LLMs via Reasoning-based Reinforcement Learning
• More Thought, Less Accuracy? On the Dual Nature of Reasoning in Vision-Language Models
• TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning
• Thinking Sparks!: Emergent Attention Heads in Reasoning Models During Post Training
• UniAPL: A Unified Adversarial Preference Learning Framework for Instruct-Following
• Scaling Generalist Data-Analytic Agents
• SemanticShield: LLM-Powered Audits Expose Shilling Attacks in Recommender Systems
• When Greedy Wins: Emergent Exploitation Bias in Meta-Bandit LLM Training
• Pushing LLMs to Their Logical Reasoning Bound: The Role of Data Reasoning Intensity
• FrameThinker: Learning to Think with Long Videos via Multi-Turn Frame Spotlighting
• Reasoning or Retrieval? A Study of Answer Attribution on Large Reasoning Models
• PIPer: On-Device Environment Setup via Online Reinforcement Learning
• Rethinking Reward Miscalibration of GRPO in Agentic RL
• Anchored Supervised Fine-Tuning
• Beyond English-Centric Training: How Reinforcement Learning Improves Cross-Lingual Reasoning in LLMs
• PaperSearchQA: Learning to Search and Reason over Scientific Papers with RLVR
• Your Models Have Thought Enough: Training Large Reasoning Models to Stop Overthinking
• Dynamic-TreeRPO: Breaking the Independent Trajectory Bottleneck with Structured Sampling
• Test-Time Policy Adaptation for Enhanced Multi-Turn Interactions with LLMs
• VideoScore2: Think before You Score in Generative Video Evaluation
• CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning
• Variational Reasoning for Language Models
• IA2: Alignment with ICL Activations Improves Supervised Fine-Tuning
• Benefits and Pitfalls of Reinforcement Learning for Language Model Planning: A Theoretical Perspective
• Rule-Based Reinforcement Learning for Document Image Classification with Vision Language Models
• The Rogue Scalpel: Activation Steering Compromises LLM Safety
• Unlocking the Essence of Beauty: Advanced Aesthetic Reasoning with Relative-Absolute Policy Optimization
• ResT: Reshaping Token-Level Policy Gradients for Tool-Use Large Language Models
• Tree Search for LLM Agent Reinforcement Learning
• Reinforcement Learning Fine-Tuning Enhances Activation Intensity and Diversity in the Internal Circuitry of LLMs
• SFT Doesn't Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMs
• RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMs
• DRES: Benchmarking LLMs for Disfluency Removal
• ThinkFake: Reasoning in Multimodal Large Language Models for AI-Generated Image Detection
• VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models
• EnAnchored-X2X: English-Anchored Optimization for Many-to-Many Translation
• UserRL: Training Interactive User-Centric Agent via Reinforcement Learning
• bi-GRPO: Bidirectional Optimization for Jailbreak Backdoor Injection on LLMs
• From Scoring to Acting: Outcome-Verified Comparative Self-Distillation for LLM Agents
• FinSMART: Financial Sentiment Analysis for Algorithmic Trading through Market-Aligned Reinforcement Learning
• Would You Walk to the Car Wash? Revealing the Salience Bias of Large Language Models in Commonsense Reasoning
• S-GRPO: Unified Post-Training for Large Vision-Language Models
• Proximal Supervised Fine-Tuning
• Reinforcement Learning on Pre-Training Data
• MAPO: Mixed Advantage Policy Optimization
• Correlation or Causation: Analyzing the Causal Structures of LLM and LRM Reasoning Process
• Generalizable End-to-End Tool-Use RL with Synthetic CodeGym
• Understanding Post-Training Structural Changes in Large Language Models
• Can GRPO Boost Complex Multimodal Table Understanding?
• Enhancing Generative Auto-bidding with Offline Reward Evaluation and Policy Search
• Compute as Teacher: Turning Inference Compute Into Reference-Free Supervision
• Explain Before You Answer: A Survey on Compositional Visual Reasoning
• Building Coding Agents via Entropy-Enhanced Multi-Turn Preference Optimization
• LaV-CoT: Language-Aware Visual CoT with Multi-Aspect Reward Optimization for Real-World Multilingual VQA
• PaVeRL-SQL: Text-to-SQL via Partial-Match Rewards and Verbal Reinforcement Learning
• Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoning
• Staying in the Sweet Spot: Responsive Reasoning Evolution via Capability-Adaptive Hint Scaffolding
• RL's Razor: Why Online Reinforcement Learning Forgets Less
• A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models
• Learning to Generate Unit Test via Adversarial Reinforcement Learning
• MedGR²: Breaking the Data Barrier for Medical Reasoning via Generative Reward Learning
• Token Buncher: Shielding LLMs from Harmful Reinforcement Learning Fine-Tuning
• AR²: Adversarial Reinforcement Learning for Abstract Reasoning in Large Language Models
• SWIRL: A Staged Workflow for Interleaved Reinforcement Learning in Mobile GUI Control
• VERIRL: Boosting the LLM-based Verilog Code Generation via Reinforcement Learning
• Toward Better EHR Reasoning in LLMs: Reinforcement Learning with Expert Attention Guidance
• ALIGN: Word Association Learning for Cultural Alignment in Large Language Models
• Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulation
• G²RPO-A: Guided Group Relative Policy Optimization with Adaptive Guidance
• Atom-Searcher: Enhancing Agentic Deep Research via Fine-Grained Atomic Thought Reward
• On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting
• UAV-VL-R1: Generalizing Vision-Language Models via Supervised Fine-Tuning and Multi-Stage GRPO for UAV Visual Reasoning
• Diversity First, Quality Later: A Two-Stage Assumption for Language Model Alignment
• Pruning Long Chain-of-Thought of Large Reasoning Models via Small-Scale Preference Optimization
• STELAR-VISION: Self-Topology-Aware Efficient Learning for Aligned Reasoning in Vision
• DocThinker: Explainable Multimodal Large Language Models with Rule-based Reinforcement Learning for Document Understanding
• Learning to Align, Aligning to Learn: A Unified Approach for Self-Optimized Alignment
• From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
• EvoCoT: Overcoming the Exploration Bottleneck in Reinforcement Learning
• Sample-efficient LLM Optimization with Reset Replay
• On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification
• Shuffle-R1: Efficient RL framework for Multimodal Large Language Models via Data-centric Dynamic Shuffle
• Decoupling Understanding from Reasoning via Problem Space Mapping for Small-Scale Model Reasoning
• GuirlVG: Incentivize GUI Visual Grounding via Empirical Exploration on Reinforcement Learning
• AttriLens-Mol: Attribute Guided Reinforcement Learning for Molecular Property Prediction with Large Language Models
• VRPRM: Process Reward Modeling via Visual Reasoning
• Beyond Surface-Level Detection: Towards Cognitive-Driven Defense Against Jailbreak Attacks via Meta-Operations Reasoning
• Tool-integrated Reinforcement Learning for Repo Deep Search
• A Rolling Stone Gathers No Moss: Adaptive Policy Optimization for Stable Self-Evaluation in Large Multimodal Models
• Reconsidering Overthinking: Penalizing Internal and External Redundancy in CoT Reasoning
• MolReasoner: Toward Effective and Interpretable Reasoning for Molecular LLMs
• SA-GCS: Semantic-Aware Gaussian Curriculum Scheduling for UAV Vision-Language Navigation
• PilotRL: Training Language Model Agents via Global Planning-Guided Progressive Reinforcement Learning

Discussions

• arxiv.org/abs/2501.17161 Yes! [bsky, 27 points, 2 comments]
• 8/ Paper: "SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training" 📝 arxiv.org/abs/2501.17161 [bsky, 5 points, 0 comments]
• Supervised Fine-Tuning Memorizes, RL Generalizes [hn, 1 points, 0 comments]
• SFT Memorizes,RL Generalizes: Comparative Study of Foundation Model PostTraining [hn, 1 points, 0 comments]
• Paper: arxiv.org/abs/2501.171... Project page: tianzhechu.com/SFTvsRL/ [bsky, 0 points, 0 comments]
• arxiv.org/abs/2501.17161 [bsky, 0 points, 1 comments]
• [Weekend Read] SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training - arxiv.org/pdf/2501.17161 - Using Reinforcement Learning (RL) help generalizing and SFT help stabil [bsky, 0 points, 0 comments]

Related

• Debunk the Myth of SFT Generalization
• Reassessing the Role of Supervised Fine-Tuning: An Empirical Study in VLM Reasoning
• Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead
• SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Models
• Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining