Kimi k1.5: Scaling Reinforcement Learning with LLMs

2025/01/22 by Angang Du, Kimi Team, Bofei Gao +173 · 5 voices · 356 citations
Engineering · Computer Science · #Robot Manipulation and Learning #Machine Learning and Data Classification #Machine Learning and Algorithms

paper · pdf · doi:10.48550/arxiv.2501.12599

Abstract

Language model pretraining with next token prediction has proved effective for scaling compute but is limited to the amount of available training data. Scaling reinforcement learning (RL) unlocks a new axis for the continued improvement of artificial intelligence, with the promise that large language models (LLMs) can scale their training data by learning to explore with rewards. However, prior published work has not produced competitive results. In light of this, we report on the training practice of Kimi k1.5, our latest multi-modal LLM trained with RL, including its RL training techniques, multi-modal data recipes, and infrastructure optimization. Long context scaling and improved policy optimization methods are key ingredients of our approach, which establishes a simplistic, effective RL framework without relying on more complex techniques such as Monte Carlo tree search, value functions, and process reward models. Notably, our system achieves state-of-the-art reasoning performance across multiple benchmarks and modalities -- e.g., 77.5 on AIME, 96.2 on MATH 500, 94-th percentile on Codeforces, 74.9 on MathVista -- matching OpenAI's o1. Moreover, we present effective long2short methods that use long-CoT techniques to improve short-CoT models, yielding state-of-the-art short-CoT reasoning results -- e.g., 60.8 on AIME, 94.6 on MATH500, 47.3 on LiveCodeBench -- outperforming existing short-CoT models such as GPT-4o and Claude Sonnet 3.5 by a large margin (up to +550%).

Cited by

• Agentic Entropy-Balanced Policy Optimization
• Reinforcement Learning via Self-Distillation
• Train for Truth, Keep the Skills: Binary Retrieval-Augmented Reward Mitigates Hallucinations
• Role-Based Fault Tolerance System for LLM RL Post-Training
• VideoZoomer: Reinforcement-Learned Temporal Focusing for Long Video Reasoning
• From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement
• AllocBench: Measuring Online Tool Allocation Capability in LLM Agents
• Progress-conditioned Group Policy Optimization for Long-Horizon Agentic Tasks
• Masked Distillation: Internalizing the Chain-of-Thought in Language Models
• LiteResearcher: A Scalable Agentic RL Training Framework for Deep Research Agent
• Anatomy-R1: Enhancing Anatomy Reasoning in Multimodal Large Language Models via Anatomical Similarity Curriculum and Group Diversity Augmentation
• ESearch-R1: Learning Cost-Aware MLLM Agents for Interactive Embodied Search via Reinforcement Learning
• Reasoning Palette: Modulating Reasoning via Latent Contextualization for Controllable Exploration for (V)LMs
• Trust-Region Adaptive Policy Optimization
• Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuning
• MMhops-R1: Multimodal Multi-hop Reasoning
• CAPE: Capability Achievement via Policy Execution
• Toward Ambulatory Vision: Learning Visually-Grounded Active View Selection
• DiG: Differential Grounding for Enhancing Fine-Grained Perception in Multimodal Large Language Model
• Boosting RL-Based Visual Reasoning with Selective Adversarial Entropy Intervention
• BAMBO: Construct Ability and Efficiency LLM Pareto Set via Bayesian Adaptive Multi-objective Block-wise Optimization
• No Labels, No Problem: Training Visual Reasoners with Multimodal Verifiers
• ShaRP: SHAllow-LayeR Pruning for Video Large Language Models Acceleration
• BioMedGPT-Mol: Multi-task Learning for Molecular Understanding and Generation
• CoRT: Code-integrated Reasoning within Thinking
• From Imitation to Discrimination: Toward A Generalized Curriculum Advantage Mechanism Enhancing Cross-Domain Reasoning Tasks
• Rectifying LLM Thought from Lens of Optimization
• Lightweight Latent Reasoning for Narrative Tasks
• G-KV: Decoding-Time KV Cache Eviction with Global Attention
• Thinking by Doing: Building Efficient World Model Reasoning in LLMs via Multi-turn Interaction
• ThetaEvolve: Test-time Learning on Open Problems
• JarvisEvo: Towards a Self-Evolving Photo Editing Agent with Synergistic Editor-Evaluator Optimization
• Revisiting the Necessity of Lengthy Chain-of-Thought in Vision-centric Reasoning Generalization
• Asking like Socrates: Socrates helps VLMs understand remote sensing images
• Guiding the Inner Eye: A Framework for Hierarchical and Flexible Visual Grounded Reasoning
• Self-Guided Defense: Adaptive Safety Alignment for Reasoning Models via Synthesized Guidelines
• Thinking in 360°: Humanoid Visual Search in the Wild
• CodeV: Code with Images for Faithful Visual Reasoning via Tool-Aware Policy Optimization
• ThreadWeaver: Adaptive Threading for Efficient Parallel Reasoning in Language Models
• DiVE-k: Differential Visual Reasoning for Fine-grained Image Recognition
• Learning to Think Fast and Slow for Visual Language Models
• An Efficient LLM-based Evolutional Recommendation with Locate-Forget-Update Paradigm
• Incorporating Self-Rewriting into Large Language Model Reasoning Reinforcement
• Fast LLM Post-training via Decoupled and Fastest-of-N Speculation
• OpenMMReasoner: Pushing the Frontiers for Multimodal Reasoning with an Open and General Recipe
• SRPO: Self-Referential Policy Optimization for Vision-Language-Action Models
• DEPO: Dual-Efficiency Preference Optimization for LLM Agents
• Empowering Multi-Turn Tool-Integrated Agentic Reasoning with Group Turn Policy Optimization
• Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learning
• Live-SWE-agent: Can Software Engineering Agents Self-Evolve on the Fly?
• Reasoning Shapes Alignment: Investigating Cultural Alignment in Large Reasoning Models with Cultural Norms
• TokenSqueeze: Performance-Preserving Compression for Reasoning LLMs
• Tailored Primitive Initialization is the Secret Key to Reinforcement Learning
• Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning
• In-Token Rationality Optimization: Towards Accurate and Concise LLM Reasoning via Self-Feedback
• From Efficiency to Adaptivity: A Deeper Look at Adaptive Reasoning in Large Language Models
• RESTL: Reinforcement Learning Guided by Multi-Aspect Rewards for Signal Temporal Logic Transformation
• Why Larger Models Learn More: Effects of Capacity, Interference, and Rare-Task Retention
• From Exploration to Exploitation: A Two-Stage Entropy RLVR Approach for Noise-Tolerant MLLM Training
• Think Consistently, Reason Efficiently: Energy-Based Calibration for Implicit Chain-of-Thought
• RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments
• What Makes Reasoning Invalid: Echo Reflection Mitigation for Large Language Models
• The Peril of Preference: Why GRPO fails on Ordinal Rewards
• CoPRIS: Efficient and Stable Reinforcement Learning via Concurrency-Controlled Partial Rollout with Importance Sampling
• Efficient Reinforcement Learning for Large Language Models with Intrinsic Exploration
• DTS: Enhancing Large Reasoning Models via Decoding Tree Sketching
• DeepCompress: A Dual Reward Strategy for Dynamically Exploring and Compressing Reasoning Chains
• Inverse Knowledge Search over Verifiable Reasoning: Synthesizing a Scientific Encyclopedia from a Long Chains-of-Thought Knowledge Base
• RoboOS-NeXT: A Unified Memory-based Framework for Lifelong, Scalable, and Robust Multi-Robot Collaboration
• e1: Learning Adaptive Control of Reasoning Effort
• Are Language Models Efficient Reasoners? A Perspective from Logic Programming
• Early Verdicts, Better Budgets: Sequential Adaptive Rollout Allocation for Compute-Efficient RLVR
• NP-Engine: Empowering Optimization Reasoning in Large Language Models with Verifiable Synthetic NP Problems
• Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability
• Sharpness-Guided Group Relative Policy Optimization via Probability Shaping
• ViPER: Empowering the Self-Evolution of Visual Perception Abilities in Vision-Language Model
• Advantage Shaping as Surrogate Reward Maximization: Unifying Pass@K Policy Gradients
• FAPO: Flawed-Aware Policy Optimization for Efficient and Reliable Reasoning
• Boosting Accuracy and Efficiency of Budget Forcing in LLMs via Reinforcement Learning for Mathematical Reasoning
• NoisyGRPO: Incentivizing Multimodal CoT Reasoning via Noise Injection and Bayesian Estimation
• MINED: Probing and Updating with Multimodal Time-Sensitive Knowledge for Large Multimodal Models
• Reasoning Language Model Inference Serving Unveiled: An Empirical Study
• UNO-Bench: A Unified Benchmark for Exploring the Compositional Law Between Uni-modal and Omni-modal in Omni Models
• Test-time Verification via Optimal Transport: Coverage, ROC, & Sub-optimality
• Explainability of Large Language Models: Opportunities and Challenges toward Generating Trustworthy Explanations
• MARSHAL: Incentivizing Multi-Agent Reasoning via Self-Play with Strategic LLMs
• LaSeR: Reinforcement Learning with Last-Token Self-Rewarding
• AI for Service: Proactive Assistance with AI Glasses
• Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanism
• The Art of Scaling Reinforcement Learning Compute for LLMs
• Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization
• Demystifying Hybrid Thinking: Can LLMs Truly Switch Between Think and No-Think?
• Laminar: A Scalable Asynchronous RL Post-Training Framework
• Reducing Belief Deviation in Reinforcement Learning for Active Reasoning
• A²FM: An Adaptive Agent Foundation Model for Tool-Aware Hybrid Reasoning
• Parallel Scaling Law: Unveiling Reasoning Generalization through A Cross-Linguistic Perspective
• Video-STR: Reinforcing MLLMs in Video Spatio-Temporal Reasoning with Relation Graph
• More Than One Teacher: Adaptive Multi-Guidance Policy Optimization for Diverse Exploration
• Cog-Rethinker: Hierarchical Metacognitive Reinforcement Learning for LLM Reasoning
• Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers
• MTSQL-R1: Towards Long-Horizon Multi-Turn Text-to-SQL via Agentic Training
• One Token Embedding Is Enough to Deadlock Your Large Reasoning Model
• Unlocking LLM Safeguards for Low-Resource Languages via Reasoning and Alignment with Minimal Training Data
• Concise Reasoning in the Lens of Lagrangian Optimization
• Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective
• SPG: Sandwiched Policy Gradient for Masked Diffusion Language Models
• Spotlight on Token Perception for Multimodal Reinforcement Learning
• Towards Efficient Multimodal Unified Reasoning Model via Model Merging
• Which Heads Matter for Reasoning? RL-Guided KV Cache Compression
• Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimization
• RePainter: Empowering E-commerce Object Removal via Spatial-matting Reinforcement Learning
• On the optimization dynamics of RLVR: Gradient gap and step size thresholds
• Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimization
• SCOPE-RL: Stable and Quantitative Control of Policy Entropy in RL Post-Training
• PEAR: Phase Entropy Aware Reward for Efficient Reasoning
• Don't Waste Mistakes: Leveraging Negative RL-Groups via Confidence Reweighting
• Entropy Regularizing Activation: Boosting Continuous Control, Large Language Models, and Image Classification with Activation as Entropy Constraints
• Pushing Test-Time Scaling Limits of Deep Search with Asymmetric Verification
• EARL: Efficient Agentic Reinforcement Learning Systems for Large Language Models
• EEPO: Exploration-Enhanced Policy Optimization via Sample-Then-Forget
• When Thinking Drifts: Evidential Grounding for Robust Video Reasoning
• Let it Calm: Exploratory Annealed Decoding for Verifiable Reinforcement Learning
• DRPO: Efficient Reasoning via Decoupled Reward Policy Optimization
• Mitigating Forgetting Between Supervised and Reinforcement Learning Yields Stronger Reasoners
• Making Mathematical Reasoning Adaptive
• Learning on the Job: Test-Time Curricula for Targeted Reinforcement Learning
• Selective Expert Guidance for Effective and Diverse Exploration in Reinforcement Learning of LLMs
• WebRenderBench: Enhancing Web Interface Generation through Layout-Style Consistency and Reinforcement Learning
• RESTRAIN: From Spurious Votes to Signals -- Self-Driven RL with Self-Penalization
• Token Hidden Reward: Steering Exploration-Exploitation in Group Relative Deep Reinforcement Learning
• Unlocking Reasoning Capabilities in LLMs via Reinforcement Learning Exploration
• Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?
• CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMs
• DecepChain: Inducing Deceptive Reasoning in Large Language Models
• Thinking-Free Policy Initialization Makes Distilled Reasoning Models More Effective and Efficient Reasoners
• RoRecomp: Enhancing Reasoning Efficiency via Rollout Response Recomposition in Reinforcement Learning
• Knapsack RL: Unlocking Exploration of LLMs via Optimizing Budget Allocation
• Overthinking Reduction with Decoupled Rewards and Curriculum Data Scheduling
• Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reuse
• Nudging the Boundaries of LLM Reasoning
• From Perception to Cognition: A Survey of Vision-Language Interactive Reasoning in Multimodal Large Language Models
• SIRI: Scaling Iterative Reinforcement Learning with Interleaved Compression
• Rethinking Entropy Regularization in Large Reasoning Models
• Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning
• AdaThink-Med: Medical Adaptive Thinking with Uncertainty-Guided Length Calibration
• Risk-Sensitive RL for Alleviating Exploration Dilemmas in Large Language Models
• Group-Relative REINFORCE Is Secretly an Off-Policy Algorithm: Demystifying Some Myths About GRPO and Its Friends
• Ultra-Fast Language Generation via Discrete Diffusion Divergence Instruct
• Emergent Slow Thinking in LLMs as Inverse Tree Freezing
• Structured In-context Environment Scaling for Large Language Model Reasoning
• Decoupling Reasoning and Perception: An LLM-LMM Framework for Faithful Visual Reasoning
• Critique to Verify: Accurate and Honest Test-Time Scaling with RL-Trained Verifiers
• Critique-Coder: Enhancing Coder Models by Critique Reinforcement Learning
• CapRL: Stimulating Dense Image Caption Capabilities via Reinforcement Learning
• Language Models Can Learn from Verbal Feedback Without Scalar Rewards
• Variational Reasoning for Language Models
• Group Critical-token Policy Optimization for Autoregressive Image Generation
• S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Models
• No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping
• Mixture-of-Visual-Thoughts: Exploring Context-Adaptive Reasoning Mode Selection for General Visual Reasoning
• Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoning
• MMR1: Enhancing Multimodal Reasoning with Variance-Aware Sampling and Open Resources
• Tree Search for LLM Agent Reinforcement Learning
• VCRL: Variance-based Curriculum Reinforcement Learning for Large Language Models
• Kalman Meets Curriculum: Efficient Dynamic Prompt Selection for Adaptive RL Finetuning
• LEEPS: Latent-Guided Explore-Exploit Prompt Sampling for Efficient RLVR in Large Language Models
• Lightning OPD 2.0: Mitigating Style Bias in Cross-Teacher On-Policy Distillation for Large Reasoning Models
• Cybersecurity Detection Classification with Reasoning-enabled Language Models
• MAPO: Mixed Advantage Policy Optimization
• AECBench: A Hierarchical Benchmark for Knowledge Evaluation of Large Language Models in the AEC Field
• NGRPO: Negative-enhanced Group Relative Policy Optimization
• Introducing LongCat-Flash-Thinking: A Technical Report
• TempSamp-R1: Effective Temporal Sampling with Reinforcement Fine-Tuning for Video LLMs
• GeoPQA: Bridging the Visual Perception Gap in MLLMs for Geometric Reasoning
• Stop Spinning Wheels: Mitigating LLM Overthinking via Mining Patterns for Early Reasoning Exit
• Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
• EMO-RL: Emotion-Rule-Based Reinforcement Learning Enhanced Audio-Language Model for Generalized Speech Emotion Recognition
• Embodied Arena: A Comprehensive, Unified, and Evolving Evaluation Platform for Embodied AI
• ToolSample: Dual Dynamic Sampling Methods with Curriculum Learning for RL-based Tool Learning
• FlowRL: Matching Reward Distributions for LLM Reasoning
• THOR: Tool-Integrated Hierarchical Optimization via RL for Mathematical Reasoning
• Single-stream Policy Optimization
• Metacognitive Reuse: Turning Recurring LLM Reasoning Into Concise Behaviors
• Reasoned Safety Alignment: Ensuring Jailbreak Defense via Answer-Then-Check
• Learning to Optimize Multi-Objective Alignment Through Dynamic Reward Weighting
• Inpainting-Guided Policy Optimization for Diffusion Large Language Models
• SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning
• VL Norm: Rethink Loss Aggregation in RLVR
• Scaling up Multi-Turn Off-Policy RL and Multi-Agent Tree Search for LLM Step-Provers
• Rethinking Reasoning Quality in Large Language Models through Enhanced Chain-of-Thought via RL
• Reverse-Engineered Reasoning for Open-Ended Generation
• From Long to Short: LLMs Excel at Trimming Own Reasoning Chains
• Simple Optimizers for Convex Aligned Multi-Objective Optimization
• Self-Aligned Reward: Towards Effective and Efficient Reasoners
• Towards a Unified View of Large Language Model Post-Training
• A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models
• Emergent Hierarchical Reasoning in LLMs through Reinforcement Learning
• Zero-shot Cross-lingual NER via Mitigating Language Difference: An Entity-aligned Translation Perspective
• Towards High Data Efficiency in Reinforcement Learning with Verifiable Reward
• PiCSAR: Probabilistic Confidence Selection And Ranking for Reasoning Chains
• Mirage or Method? How Model-Task Alignment Induces Divergent RL Conclusions
• Intern-S1: A Scientific Multimodal Foundation Model
• rStar2-Agent: Agentic Reasoning Technical Report
• Think in Blocks: Adaptive Reasoning from Direct Response to Deep Reasoning
• Demystifying Scientific Problem-Solving in LLMs by Probing Knowledge and Reasoning
• ThinkDial: An Open Recipe for Controlling Reasoning Effort in Large Language Models
• Recall-Extend Dynamics: Enhancing Small Language Models through Controlled Exploration and Refined Offline Integration
• Deep Think with Confidence
• LeanGeo: Formalizing Competitional Geometry problems in Lean
• DuPO: Enabling Reliable LLM Self-Verification via Dual Preference Optimization
• Depth-Breadth Synergy in RLVR: Unlocking LLM Reasoning Gains with Adaptive Exploration
• Embodied-R1: Reinforced Embodied Reasoning for General Robotic Manipulation
• Atom-Searcher: Enhancing Agentic Deep Research via Fine-Grained Atomic Thought Reward
• Reinforcement Learning with Rubric Anchors
• Simple o3: Towards Interleaved Vision-Language Reasoning
• Aware First, Think Less: Dynamic Boundary Self-Awareness Drives Extreme Reasoning Efficiency in Large Language Models
• SeamlessFlow: A Trainer Agent Isolation RL Framework Achieving Bubble-Free Pipelines via Tag Scheduling
• CRAFT-GUI: Curriculum-Reinforced Agent For GUI Tasks
• CURE: Critical-Token-Guided Re-Concatenation for Entropy-Collapse Prevention
• Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Models
• Pruning Long Chain-of-Thought of Large Reasoning Models via Small-Scale Preference Optimization
• MEML-GRPO: Heterogeneous Multi-Expert Mutual Learning for RLVR Advancement
• Audio-Thinker: Guiding Audio Language Model When and How to Think via Reinforcement Learning
• EvoCoT: Overcoming the Exploration Bottleneck in Reinforcement Learning
• AMFT: Aligning LLM Reasoners by Meta-Learning the Optimal Imitation-Exploration Balance
• SABER: Switchable and Balanced Training for Efficient LLM Reasoning
• Pruning the Unsurprising: Efficient LLM Reasoning via First-Token Surprisal
• Shuffle-R1: Efficient RL framework for Multimodal Large Language Models via Data-centric Dynamic Shuffle
• StructVRM: Aligning Multimodal Reasoning with Structured and Verifiable Reward Models
• FedCoT: Communication-Efficient Federated Reasoning Enhancement for Large Language Models
• Decoupling Understanding from Reasoning via Problem Space Mapping for Small-Scale Model Reasoning
• FinMMR: Make Financial Numerical Reasoning More Multimodal, Comprehensive, and Challenging
• StepFun-Formalizer: Unlocking the Autoformalization Potential of LLMs through Knowledge-Reasoning Fusion
• ReasoningGuard: Safeguarding Large Reasoning Models with Inference-time Safety Aha Moments
• Agent Lightning: Train ANY AI Agents with Reinforcement Learning
• Towards Trustworthy Multimodal Moderation via Policy-Aligned Reasoning and Hierarchical Labeling
• An Efficient and Adaptive Next Edit Suggestion Framework with Zero Human Instructions in IDEs
• Test-time Prompt Intervention
• MedVLThinker: Simple Baselines for Multimodal Medical Reasoning
• Reconsidering Overthinking: Penalizing Internal and External Redundancy in CoT Reasoning
• PilotRL: Training Language Model Agents via Global Planning-Guided Progressive Reinforcement Learning
• RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization
• GraphRAG-R1: Graph Retrieval-Augmented Generation with Process-Constrained Reinforcement Learning
• Stabilizing Knowledge, Promoting Reasoning: Dual-Token Constraints for RLVR
• Good Learners Think Their Thinking: Generative PRM Makes Large Reasoning Model More Efficient Math Learner
• VL-Cogito: Progressive Curriculum Reinforcement Learning for Advanced Multimodal Reasoning
• ChemDFM-R: A Chemical Reasoning LLM Enhanced with Atomized Chemical Knowledge
• EDGE-GRPO: Entropy-Driven GRPO with Guided Error Correction for Advantage Diversity
• Few-Shot Vision-Language Reasoning for Satellite Imagery via Verifiable Rewards
• Libra: Assessing and Improving Reward Model by Learning to Think
• Learning Only with Images: Visual Reinforcement Learning with Reasoning, Rendering, and Visual Feedback
• Kimi K2: Open Agentic Intelligence
• A Survey of Token Compression for Efficient Multimodal Large Language Models
• Agentic Reinforced Policy Optimization
• UloRL:An Ultra-Long Output Reinforcement Learning Approach for Advancing Large Language Models' Reasoning Abilities
• MindSpeed RL: Distributed Dataflow for Scalable and Efficient RL Training on Ascend NPU Cluster
• Decoupling Knowledge and Reasoning in LLMs: An Exploration Using Cognitive Dual-System Theory
• Prune&Comp: Free Lunch for Layer-Pruned LLMs via Iterative Pruning with Magnitude Compensation
• AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning
• MiroMind-M1: An Open-Source Advancement in Mathematical Reasoning via Context-Aware Multi-Stage Policy Optimization
• BusterX++: Towards Unified Cross-Modal AI-Generated Content Detection and Explanation with MLLM
• KROMA: Ontology Matching with Knowledge Retrieval and Large Language Models
• INTEGRALBENCH: Benchmarking LLMs with Definite Integral Problems
• VAR-MATH: Probing True Mathematical Reasoning in LLMS via Symbolic Multi-Instance Benchmarks
• ROC-n-reroll: How verifier imperfection affects test-time scaling
• Let's Think in Two Steps: Mitigating Agreement Bias in MLLMs with Self-Grounded Verification
• SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learning
• Learning to Reason Across Parallel Samples for LLM Reasoning
• SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning
• A Survey on Large Language Models for Mathematical Reasoning
• REST: Stress Testing Large Reasoning Models by Asking Multiple Problems at Once
• GHPO: Adaptive Guidance for Stable and Efficient LLM Reinforcement Learning
• VerifyBench: A Systematic Benchmark for Evaluating Reasoning Verifiers Across Domains
• DeepSeek: Paradigm Shifts and Technical Evolution in Large AI Models
• Towards Concise and Adaptive Thinking in Large Reasoning Models: A Survey
• Video-CoT: A Comprehensive Dataset for Spatiotemporal Understanding of Videos Based on Chain-of-Thought
• How to Train a Leader: Hierarchical Reasoning in Multi-Agent LLMs
• One Token to Fool LLM-as-a-Judge
• Leanabell-Prover-V2: Verifier-integrated Reasoning for Formal Theorem Proving via Reinforcement Learning
• M2-Reasoning: Empowering MLLMs with Unified General and Spatial Reasoning
• MIRA: A Novel Framework for Fusing Modalities in Medical RAG
• Scaling RL to Long Videos
• The Synergy Dilemma of Long-CoT SFT and RL: Investigating Post-Training Techniques for Reasoning VLMs
• Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Model
• From Data-Centric to Sample-Centric: Enhancing LLM Reasoning via Progressive Optimization
• Perception-Aware Policy Optimization for Multimodal Reasoning
• CoRE: Enhancing Metacognition with Label-free Self-evaluation in LRMs
• BlueLM-2.5-3B Technical Report
• Open Vision Reasoner: Transferring Linguistic Cognitive Behavior for Visual Reasoning
• Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?
• Pre-Trained Policy Discriminators are General Reward Models
• A Technical Survey of Reinforcement Learning Techniques for Large Language Models
• APPO: Agentic Procedural Policy Optimization
• Multimodal Mathematical Reasoning with Diverse Solving Perspective
• ExPO: Unlocking Hard Reasoning with Self-Explanation-Guided Reinforcement Learning
• TimeRFT: Stimulating Generalizable Time Series Forecasting for TSFMs via Reinforcement Finetuning
• Think How to Think: Mitigating Overthinking with Autonomous Difficulty Cognition in Large Reasoning Models
• MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agent
• Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMs
• AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training
• Improving the Reasoning of Multi-Image Grounding in MLLMs via Reinforcement Learning
• Reasoning as an Adaptive Defense for Safety
• ASTRO: Teaching Language Models to Reason by Reflecting and Backtracking In-Context
• MMReason: An Open-Ended Multi-Modal Multi-Step Reasoning Benchmark for MLLMs Toward AGI
• Do Thinking Tokens Help or Trap? Towards More Efficient Large Reasoning Model
• EFRame: Deeper Reasoning via Exploration-Filter-Replay Reinforcement Learning Framework
• MiCo: Multi-image Contrast for Reinforcement Visual Reasoning
• Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions
• DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPO
• Double-Checker: Enhancing Reasoning of Slow-Thinking LLMs via Self-Critical Fine-Tuning
• MMSearch-R1: Incentivizing LMMs to Search
• EAR: Erasing Concepts from Unified Autoregressive Models
• ReCode: Updating Code API Knowledge with Reinforcement Learning
• Enhancing Large Language Models through Structured Reasoning
• Scaling Speculative Decoding with Lookahead Reasoning
• Is Long-to-Short a Free Lunch? Investigating Inconsistency and Reasoning Efficiency in LRMs
• Skywork-SWE: Unveiling Data Scaling Laws for Software Engineering in LLMs
• LongWriter-Zero: Mastering Ultra-Long Text Generation via Reinforcement Learning
• Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning
• A Simple "Motivation" Can Enhance Reinforcement Finetuning of Large Reasoning Models
• WeThink: Toward General-purpose Vision-Language Reasoning via Reinforcement Learning
• BREAD: Branched Rollouts from Expert Anchors Bridge SFT & RL for Reasoning
• Chiron-o1: Igniting Multimodal Large Language Models towards Generalizable Medical Reasoning via Mentor-Intern Collaborative Search
• No Free Lunch: Rethinking Internal Feedback for LLM Reasoning
• VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuning
• From Thinking to Output: Chain-of-Thought and Text Generation Characteristics in Reasoning Language Models
• OJBench: A Competition Level Code Benchmark For Large Language Models
• GRPO-CARE: Consistency-Aware Reinforcement Learning for Multimodal Reasoning
• Steering Your Diffusion Policy with Latent Space Reinforcement Learning
• Exploring and Exploiting the Inherent Efficiency within Large Reasoning Models for Self-Guided Efficiency Enhancement
• ProtoReasoning: Prototypes as the Foundation for Generalizable Reasoning in LLMs
• Truncated Proximal Policy Optimization
• Fractional Reasoning via Latent Steering Vectors Improves Inference Time Compute
• Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Models
• Perception-R1: Advancing Multimodal Reasoning Capabilities of MLLMs via Visual Perception Reward
• TimeMaster: Training Time-Series Multimodal LLMs to Reason via Reinforcement Learning
• MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attention
• Socratic RL: A Novel Framework for Efficient Knowledge Acquisition through Iterative Reflection and Viewpoint Distillation
• QFFT, Question-Free Fine-Tuning for Adaptive Reasoning
• Reasoning Model Unlearning: Forgetting Traces, Not Just Answers, While Preserving Reasoning Skills
• Perspective on Utilizing Foundation Models for Laboratory Automation in Materials Research
• Efficient Reasoning Through Suppression of Self-Affirmation Reflections in Large Reasoning Models
• Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuning
• How Far Are We from Optimal Reasoning Efficiency?
• From Emergence to Control: Probing and Modulating Self-Reflection in Language Models
• VFaith: Do Large Multimodal Models Really Reason on Seen Images Rather than Previous Memories?
• What makes Reasoning Models Different? Follow the Reasoning Leader for Efficient Decoding
• Spurious Rewards: Rethinking Training Signals in RLVR
• AdaptiveLLM: A Framework for Selecting Optimal Cost-Efficient LLM for Code-Generation Based on CoT Length
• LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learning
• Causal Sufficiency and Necessity Improves Chain-of-Thought Reasoning
• Athena: Enhancing Multimodal Reasoning with Data-efficient Process Reward Models
• Table-r1: Self-supervised and Reinforcement Learning for Program-based Table Reasoning in Small Language Models
• SPRINT: Enabling Interleaved Planning and Parallelized Execution in Reasoning Models
• Can Theoretical Physics Research Benefit from Language Agents?
• Sample Complexity and Representation Ability of Test-time Scaling Paradigms

Discussions

• Kimi K1.5 Technical Report [hn, 4 points, 0 comments]
• Kimi K1.5: Scaling Reinforcement Learning with LLMs [hn, 4 points, 0 comments]
• 1.5 came out alongside R1 right? I haven't used it or skimmed their paper yet--does anyone have any opinions on Kimi 1.5? Aforementioned paper: arxiv.org/abs/2501.12599 [bsky, 1 points, 1 comments]
• arxiv.org/abs/2501.12599 [bsky, 0 points, 0 comments]
• In the wake of the Kimi K2 release, a look back at the foundational changes made in Kimi K1.5 for dynamic RL techniques over extended context windows. arxiv.org/abs/2501.12599 [bsky, 0 points, 0 comments]

Related

• Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
• The Art of Scaling Reinforcement Learning Compute for LLMs
• Reinforcement Learning on Pre-Training Data
• SRPO: A Cross-Domain Implementation of Large-Scale Reinforcement Learning on LLM
• Scaling Behaviors of LLM Reinforcement Learning Post-Training: An Empirical Study in Mathematical Reasoning