Reinforcement Learning for Large Model: A Survey

2025/08/11 by Wu, Weijia, Gao, Chen, Chen, Joya +6 · 1 citation
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2508.08189

Abstract

Recent advances at the intersection of reinforcement learning (RL) and visual intelligence have enabled agents that not only perceive complex visual scenes but also reason, generate, and act within them. This survey offers a critical and up-to-date synthesis of the field. We first formalize visual RL problems and trace the evolution of policy-optimization strategies from RLHF to verifiable reward paradigms, and from Proximal Policy Optimization to Group Relative Policy Optimization. We then organize more than 200 representative works into four thematic pillars: multi-modal large language models, visual generation, unified model frameworks, and vision-language-action models. For each pillar we examine algorithmic design, reward engineering, benchmark progress, and we distill trends such as curriculum-driven training, preference-aligned diffusion, and unified reward modeling. Finally, we review evaluation protocols spanning set-level fidelity, sample-level preference, and state-level stability, and we identify open challenges that include sample efficiency, generalization, and safe deployment. Our goal is to provide researchers and practitioners with a coherent map of the rapidly expanding landscape of visual RL and to highlight promising directions for future inquiry. Resources are available at: https://github.com/weijiawu/Awesome-Visual-Reinforcement-Learning.

Citations

• MIRO: MultI-Reward cOnditioned pretraining improves T2I quality and efficiency
• Sample By Step, Optimize By Chunk: Chunk-Level GRPO For Text-to-Image Generation
• CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning
• MOSS-ChatV: Reinforcement Learning with Process Reasoning Reward for Video Temporal Reasoning
• TempSamp-R1: Effective Temporal Sampling with Reinforcement Fine-Tuning for Video LLMs
• THOR: Tool-Integrated Hierarchical Optimization via RL for Mathematical Reasoning
• OneReward: Unified Mask-Guided Image Generation via Multi-Task Human Preference Learning
• Video-MTR: Reinforced Multi-Turn Reasoning for Long Video Understanding
• Pref-GRPO: Pairwise Preference Reward-based GRPO for Stable Text-to-Image Reinforcement Learning
• USO: Unified Style and Subject-Driven Generation via Disentangled and Reward Learning
• LENS: Learning to Segment Anything with Unified Reinforced Reasoning
• BigCharts-R1: Enhanced Chart Reasoning with Visual Reinforcement Finetuning
• DocR1: Evidence Page-Guided GRPO for Multi-Page Document Understanding
• IRL-VLA: Training an Vision-Language-Action Policy via Reward World Model
• TempFlow-GRPO: When Timing Matters for GRPO in Flow Models
• Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoning
• X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Again
• Multimodal LLMs as Customized Reward Models for Text-to-Image Generation
• Learning Only with Images: Visual Reinforcement Learning with Reasoning, Rendering, and Visual Feedback
• VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learning
• Inversion-DPO: Precise and Efficient Post-Training for Diffusion Models
• Scaling RL to Long Videos
• GTA1: GUI Test-time Scaling Agent
• MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environment
• Improving the Reasoning of Multi-Image Grounding in MLLMs via Reinforcement Learning
• HumanOmniV2: From Understanding to Omni-Modal Reasoning with Context
• Mobile-R1: Towards Interactive Reinforcement Learning for VLM-Based Mobile Agent via Task-Level Rewards
• VQ-Insight: Teaching VLMs for AI-Generated Video Quality Understanding via Progressive Visual Reinforcement Learning
• RePIC: Reinforced Post-Training for Personalizing Multi-Modal Language Models
• Drive-R1: Bridging Reasoning and Planning in VLMs for Autonomous Driving with Reinforcement Learning
• Scene-R1: Video-Grounded Large Language Models for 3D Scene Reasoning without 3D Annotations
• VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuning
• GRPO-CARE: Consistency-Aware Reinforcement Learning for Multimodal Reasoning
• GoalLadder: Incremental Goal Discovery with Vision-Language Models
• OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agents
• Nabla-R2D3: Effective and Efficient 3D Diffusion Alignment with 2D Rewards
• VL-GenRM: Enhancing Vision-Language Verification via Vision Experts and Iterative Training
• TimeMaster: Training Time-Series Multimodal LLMs to Reason via Reinforcement Learning
• Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoning
• EasyARC: Evaluating Vision Language Models on True Visual Reasoning
• Reinforcing Spatial Reasoning in Vision-Language Models with Interwoven Thinking and Visual Drawing
• LPO: Towards Accurate GUI Agent Interaction via Location Preference Optimization
• ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs
• DreamCS: Geometry-Aware Text-to-3D Generation with Unpaired 3D Reward Supervision
• OctoNav: Towards Generalist Embodied Navigation
• TGRPO :Fine-tuning Vision-Language-Action Model via Trajectory-wise Group Relative Policy Optimization
• VIKI-R: Coordinating Embodied Multi-Agent Cooperation via Reinforcement Learning
• DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPO
• WeThink: Toward General-purpose Vision-Language Reasoning via Reinforcement Learning
• FocusDiff: Advancing Fine-Grained Text-Image Alignment for Autoregressive Visual Generation through RL
• MiMo-VL Technical Report
• Q-Ponder: A Unified Training Pipeline for Reasoning-based Visual Quality Assessment
• EgoVLM: Policy Optimization for Egocentric Video Understanding
• SVQA-R1: Reinforcing Spatial Reasoning in MLLMs via View-Consistent Reward Optimization
• AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuning
• Reinforcement Learning Tuning for VideoLLMs: Reward Design and Data Efficiency
• SRPO: Enhancing Multimodal LLM Reasoning via Reflection-Aware Reinforcement Learning
• GThinker: Towards General Multimodal Reasoning via Cue-Guided Rethinking
• ReasonGen-R1: CoT for Autoregressive Image generation models through SFT and RL
• Reinforcing Video Reasoning with Focused Thinking
• ProxyThinker: Test-Time Guidance through Small Visual Reasoners
• MoDoMoDo: Multi-Domain Data Mixtures for Multimodal LLM Reinforcement Learning
• UniRL: Self-Improving Unified Multimodal Models via Supervised and Reinforcement Learning
• Grounded Reinforcement Learning for Visual Reasoning
• DIP-R1: Deep Inspection and Perception with RL Looking Through and Understanding Complex Scenes
• Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Robotics
• VAU-R1: Advancing Video Anomaly Understanding via Reinforcement Fine-Tuning
• Qwen Look Again: Guiding Vision-Language Reasoning Models to Re-attention Visual Information
• Fine-Tuning Next-Scale Visual Autoregressive Models with Group Relative Policy Optimization
• Jigsaw-R1: A Study of Rule-based Visual Reinforcement Learning with Jigsaw Puzzles
• Thinking with Generated Images
• D-Fusion: Direct Preference Optimization for Aligning Diffusion Models with Visually Consistent Samples
• VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learning
• Rendering-Aware Reinforcement Learning for Vector Graphics Generation
• MME-Reasoning: A Comprehensive Benchmark for Logical Reasoning in MLLMs
• Active-O3: Empowering Multimodal Large Language Models with Active Perception via GRPO
• Video-Holmes: Can MLLM Think Like Holmes for Complex Video Reasoning?
• TACO: Think-Answer Consistency for Optimized Long-Chain Reasoning and Efficient Data Learning via Reinforcement Learning in LVLMs
• TeViR: Text-to-Video Reward with Diffusion Models for Efficient Reinforcement Learning
• RFTF: Reinforcement Fine-tuning for Embodied Agents with Temporal Feedback
• What Can RL Bring to VLA Generalization? An Empirical Study
• Omni-R1: Reinforcement Learning for Omnimodal Reasoning via Two-System Collaboration
• Ground-R1: Incentivizing Grounded Visual Reasoning via Reinforcement Learning
• VisualToolAgent (VisTA): A Reinforcement Learning Framework for Visual Tool Selection
• VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning
• Enhancing LLMs' Reasoning-Intensive Multimedia Search Capabilities through Fine-Tuning and Reinforcement Learning
• Co-Reinforcement Learning for Unified Multimodal Understanding and Generation
• InfLVG: Reinforce Inference-Time Consistent Long Video Generation with GRPO
• ProgRM: Build Better GUI Agents with Progress Rewards
• One RL to See Them All: Visual Triple Unified Reinforcement Learning
• RePrompt: Reasoning-Augmented Reprompting for Text-to-Image Generation via Reinforcement Learning
• ARPO:End-to-End Policy Optimization for GUI Agents with Experience Replay
• GoT-R1: Unleashing Reasoning Capability of MLLM for Visual Generation with Reinforcement Learning
• Interactive Post-Training for Vision-Language-Action Models
• Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Models
• Self-Rewarding Large Vision-Language Models for Optimizing Prompts in Text-to-Image Generation
• R1-ShareVL: Incentivizing Reasoning Capability of Multimodal Large Language Models via Share-GRPO
• STAR-R1: Spatial TrAnsformation Reasoning by Reinforcing Multimodal LLMs
• VARD: Efficient and Dense Fine-Tuning for Diffusion Models with Value-based RL
• GRIT: Teaching MLLMs to Think with Images
• Pixel Reasoner: Incentivizing Pixel-Space Reasoning with Curiosity-Driven Reinforcement Learning
• DeepEyes: Incentivizing "Thinking with Images" via Reinforcement Learning
• Visionary-R1: Mitigating Shortcuts in Visual Reasoning with Reinforcement Learning
• UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learning
• VisualQuality-R1: Reasoning-Induced Image Quality Assessment via Reinforcement Learning to Rank
• G1: Bootstrapping Perception and Reasoning Abilities of Vision-Language Model via Reinforcement Learning
• Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning
• Observe-R1: Unlocking Reasoning Abilities of MLLMs with Dynamic Progressive Reinforcement Learning
• Visual Planning: Let's Think Only with Images
• OpenThinkIMG: Learning to Think with Images via Visual Tool Reinforcement Learning
• Selftok: Discrete Visual Tokens of Autoregression, by Diffusion, and for Reasoning
• DanceGRPO: Unleashing GRPO on Visual Generation
• ReinboT: Amplifying Robot Visual-Language Manipulation with Reinforcement Learning
• Flow-GRPO: Training Flow Matching Models via Online RL
• EchoInk-R1: Exploring Audio-Visual Reasoning in Multimodal LLMs via Reinforcement Learning
• Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuning
• Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Models
• Skywork R1V2: Multimodal Hybrid Reinforcement Learning for Reasoning
• Reasoning Physical Video Generation with Diffusion Timestep Tokens via Reinforcement Learning
• VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models
• DSPO: Direct Semantic Preference Optimization for Real-World Image Super-Resolution
• Embodied-R: Collaborative Framework for Activating Embodied Spatial Reasoning in Foundation Models via Reinforcement Learning
• SimpleAR: Pushing the Frontier of Autoregressive Visual Generation through Pretraining, SFT, and RL
• GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents
• Aligning Anime Video Generation with Human Feedback
• AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories
• Perception-R1: Pioneering Perception Policy with Reinforcement Learning
• VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model
• VideoChat-R1: Enhancing Spatio-Temporal Perception via Reinforcement Fine-Tuning
• ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use
• VARGPT-v1.1: Improve Visual Autoregressive Large Unified Model via Iterative Instruction Tuning and Reinforcement Learning
• Exploring the Effect of Reinforcement Learning on Video Understanding: Insights from SEED-Bench-R1
• UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learning
• Video-R1: Reinforcing Video Reasoning in MLLMs
• MetaSpatial: Reinforcing 3D Spatial Reasoning in VLMs for the Metaverse
• Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning
• UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction
• Time-R1: Post-Training Large Vision Language Model for Temporal Video Grounding
• Towards Better Alignment: Training Diffusion Models with Reinforcement Learning Against Sparse Rewards
• R1-Onevision: Advancing Generalized Multimodal Reasoning through Cross-Modal Formalization
• MoRE: Unlocking Scalability in Reinforcement Learning for Quadruped Vision-Language-Action Models
• SegAgent: Exploring Pixel Understanding Capabilities in MLLMs by Imitating Human Annotator Trajectories
• VisRL: Intention-Driven Visual Perception via Reinforced Reasoning
• MM-Eureka: Exploring the Frontiers of Multimodal Reasoning with Rule-based Reinforcement Learning
• Unified Reward Model for Multimodal Understanding and Generation
• GRADEO: Towards Human-Like Evaluation for Text-to-Video Generation via Multi-Step Reasoning
• Visual-RFT: Visual Reinforcement Fine-Tuning
• A Simple and Effective Reinforcement Learning Method for Text-to-Image Diffusion Fine-tuning
• Multimodal Inconsistency Reasoning (MMIR): A New Benchmark for Multimodal Reasoning Models
• HermesFlow: Seamlessly Closing the Gap in Multimodal Understanding and Generation
• MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
• ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models
• AppVLM: A Lightweight Vision Language Model for Online App Control
• ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policy
• DreamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization
• RAPID: Robust and Agile Planner Using Inverse Reinforcement Learning for Vision-Based Drone Navigation
• MM-IQ: Benchmarking Human-Like Abstraction and Reasoning in Multimodal Models
• Improving Video Generation with Human Feedback
• Can We Generate Images with CoT? Let's Verify and Reinforce Image Generation Step by Step
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• UI-TARS: Pioneering Automated GUI Interaction with Native Agents
• Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark
• VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks
• Thinking in Space: How Multimodal Large Language Models See, Remember, and Recall Spaces
• RLDG: Robotic Generalist Policy Distillation via Reinforcement Learning
• Optical-Flow Guided Prompt Optimization for Coherent Video Generation
• Tulu 3: Pushing Frontiers in Open Language Model Post-Training
• Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning
• PrefPaint: Aligning Image Inpainting Diffusion Model with Human Preference
• Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation
• Emu3: Next-Token Prediction is All You Need
• FLaRe: Achieving Masterful and Adaptive Robot Policies with Large-Scale Reinforcement Learning Fine-Tuning
• Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale
• MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark
• Subject-driven Text-to-Image Generation via Preference-based Reinforcement Learning
• Powerful and Flexible: Personalized Text-to-Image Generation via Reinforcement Learning
• We-Math: Does Your Large Multimodal Model Achieve Human-like Mathematical Reasoning?
• LLM Critics Help Catch LLM Bugs
• VideoScore: Building Automatic Metrics to Simulate Fine-grained Human Feedback for Video Generation
• Is A Picture Worth A Thousand Words? Delving Into Spatial Reasoning for Vision Language Models
• DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning
• InstructRL4Pix: Training Diffusion for Image Editing by Reinforcement Learning
• Diffusion-RPO: Aligning Diffusion Models through Relative Preference Optimization
• RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots
• OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
• Aligning Diffusion Models by Optimizing Human Utility
• DreamReward: Text-to-3D Generation with Human Preference
• MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?
• ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment
• Android in the Zoo: Chain-of-Action-Thought for GUI Agents
• OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web
• OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning
• CMMMU: A Chinese Massive Multi-discipline Multimodal Understanding Benchmark
• SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents
• Parrot: Pareto-optimal Multi-Reward Reinforcement Learning Framework for Text-to-Image Generation
• InstructVideo: Instructing Video Diffusion Models with Human Feedback
• StarVector: Generating Scalable Vector Graphics Code from Images and Text
• Rich Human Feedback for Text-to-Image Generation
• Enhancing Diffusion Models with Text-Encoder Reinforcement Learning
• Diffusion Model Alignment Using Direct Preference Optimization
• Selective Visual Representations Improve Convergence and Generalization for Embodied AI
• Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots
• GenEval: An Object-Focused Framework for Evaluating Text-to-Image Alignment
• Reinforced UI Instruction Grounding: Towards a Generic UI Task Automation API
• MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
• Aligning Large Multimodal Models with Factually Augmented RLHF
• Autonomous Tester Agent Benchmark
• MMBench: Is Your Multi-modal Model an All-around Player?
• MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models
• Human Preference Score v2: A Solid Benchmark for Evaluating Human Preferences of Text-to-Image Synthesis
• Mind2Web: Towards a Generalist Agent for the Web
• LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• DPOK: Reinforcement Learning for Fine-tuning Text-to-Image Diffusion Models
• Training Diffusion Models with Reinforcement Learning
• Evaluating Object Hallucination in Large Vision-Language Models
• Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation
• ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation
• Human Preference Score: Better Aligning Text-to-Image Models with Human Preference
• MoDem: Accelerating Visual Model-Based Reinforcement Learning with Demonstrations
• CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning
• A Study on the Evaluation of Generative Models
• Training language models to follow instructions with human feedback
• Deep Reinforcement Learning at the Edge of the Statistical Precipice
• InfographicVQA
• Learning Transferable Visual Models From Natural Language Supervision
• DocVQA: A Dataset for VQA on Document Images
• Evaluating the Performance of Reinforcement Learning Algorithms
• Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous\n Environments
• RLBench: The Robot Learning Benchmark & Learning Environment
• Towards VQA Models That Can Read
• Generalized Intersection over Union: A Metric and A Loss for Bounding\n Box Regression
• Learning Synergies between Pushing and Grasping with Self-supervised Deep Reinforcement Learning
• Proximal Policy Optimization Algorithms
• Deep reinforcement learning from human preferences
• Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering
• Improved Techniques for Training GANs
• High-Dimensional Continuous Control Using Generalized Advantage Estimation
• Image quality assessment: from error visibility to structural similarity
• GUI-Shift: Enhancing VLM-Based GUI Agents through Self-supervised Reinforcement Learning
• VisionReasoner: Unified Reasoning-Integrated Visual Perception via Reinforcement Learning
• Reason-RFT: Reinforcement Fine-Tuning for Visual Reasoning of Vision Language Models
• OpenAI o1 System Card
• Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning
• T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generation

Cited by

• Self-Improvement in Multimodal Large Language Models: A Survey

Related

• Aligning Large Vision-Language Models by Deep Reinforcement Learning and Direct Preference Optimization
• Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning
• RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback
• VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model
• Reinforcement Learning Enhanced LLMs: A Survey