Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learning

2025/11/18 by Qin, Ruoyu, He, Weiran, Huang, Weixiao +7 · 2 citations
#Distributed #FOS: Computer and information sciences #Machine Learning (cs.LG) #Parallel #and Cluster Computing (cs.DC)

paper · doi:10.48550/arxiv.2511.14617

Abstract

Reinforcement Learning (RL) has become critical for advancing modern Large Language Models (LLMs), yet existing synchronous RL systems face severe performance bottlenecks. The rollout phase, which dominates end-to-end iteration time, suffers from substantial long-tail latency and poor resource utilization due to inherent workload imbalance. We present Seer, a novel online context learning system that addresses these challenges by exploiting previously overlooked similarities in output lengths and generation patterns among requests sharing the same prompt. Seer introduces three key techniques: divided rollout for dynamic load balancing, context-aware scheduling, and adaptive grouped speculative decoding. Together, these mechanisms substantially reduce long-tail latency and improve resource efficiency during rollout. Evaluations on production-grade RL workloads demonstrate that Seer improves end-to-end rollout throughput by 74% to 97% and reduces long-tail latency by 75% to 93% compared to state-of-the-art synchronous RL systems, significantly accelerating RL training iterations.

Citations

• Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
• RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMs
• Laminar: A Scalable Asynchronous RL Post-Training Framework
• Part II: ROLL Flash -- Accelerating RLVR and Agentic Training with Asynchrony
• BroRL: Scaling Reinforcement Learning via Broadened Exploration
• RollPacker: Mitigating Long-Tail Rollouts for Fast, Synchronous RL Post-Training
• PipelineRL: Faster On-policy Reinforcement Learning for Long Sequence Generation
• APRIL: Active Partial Rollouts in Reinforcement Learning to Tame Long-tail Generation
• Introducing LongCat-Flash-Thinking: A Technical Report
• History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRL
• Group Sequence Policy Optimization
• AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training
• Reinforcement Learning Optimization for Large-Scale Learning: An Efficient and User-Friendly Scaling Library
• AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning
• StreamRL: Scalable, Heterogeneous, and Elastic RL for LLMs with Disaggregated Stream Generation
• DAPO: An Open-Source LLM Reinforcement Learning System at Scale
• EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test
• Muon is Scalable for LLM Training
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Kimi k1.5: Scaling Reinforcement Learning with LLMs
• SAM Decoding: Speculative Decoding via Suffix Automaton
• SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications
• Optimizing RLHF Training for Large Language Models with Stage Fusion
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• LLM-as-a-Judge & Reward Model: What They Can and Cannot Do
• Efficient LLM Scheduling by Learning to Rank
• EAGLE-2: Faster Inference of Language Models with Dynamic Draft Trees
• ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation
• Enabling Efficient Batch Serving for LMaaS via Generation Length Prediction
• Llumnix: Dynamic Scheduling for Large Language Model Serving
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
• NeMo-Aligner: Scalable Toolkit for Efficient Model Alignment
• OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
• EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty
• Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads
• SGLang: Efficient Execution of Structured Language Model Programs
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
• Fast Inference from Transformers via Speculative Decoding
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Evaluating Large Language Models Trained on Code
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Rate control for communication networks: shadow prices, proportional fairness and stability
• SPEC-RL: Accelerating On-Policy Reinforcement Learning via Speculative Rollouts
• OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework

Cited by

• RollMux: Phase-Level Multiplexing for Disaggregated RL Post-Training
• Fast LLM Post-training via Decoupled and Fastest-of-N Speculation

Related

• History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRL
• DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Training
• AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Training
• DistFlow: A Fully Distributed RL Framework for Scalable and Efficient LLM Post-Training
• RLBoost: Harvesting Preemptible Resources for Cost-Efficient Reinforcement Learning on LLMs