G-Core: A Simple, Scalable and Balanced RLHF Trainer

2025/07/30 by Junyu Wu, Wu, Junyu, Chang, Weiming +15 · 1 citation
Engineering · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Real-time simulation and control systems

paper · pdf · doi:10.48550/arxiv.2507.22789

openalex publication_date 2025/07/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Reinforcement Learning from Human Feedback (RLHF) has become an increasingly popular paradigm for training large language models (LLMs) and diffusion models. While existing RLHF training systems have enabled significant progress, they often face challenges in scaling to multi-modal and diffusion workflows and adapting to dynamic workloads. In particular, current approaches may encounter limitations in controller scalability, flexible resource placement, and efficient orchestration when handling complex RLHF pipelines, especially in scenarios involving dynamic sampling or generative reward modeling. In this paper, we present G-Core, a simple, scalable, and balanced RLHF training framework designed to address these challenges. G-Core introduces a parallel controller programming model, enabling flexible and efficient orchestration of complex RLHF workflows without the bottlenecks of a single centralized controller. Furthermore, we propose a dynamic placement schema that adaptively partitions resources and schedules workloads, significantly reducing hardware idle time and improving utilization, even under highly variable training conditions. G-Core has successfully trained models that support WeChat product features serving a large-scale user base, demonstrating its effectiveness and robustness in real-world scenarios. Our results show that G-Core advances the state of the art in RLHF training, providing a solid foundation for future research and deployment of large-scale, human-aligned models.

Citations

• AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning
• MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Core
• Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement
• Generative Verifiers: Reward Modeling as Next-Token Prediction
• An Adaptive Placement and Parallelism Framework for Accelerating RLHF Training
• ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models
• Ring Attention with Blockwise Transformers for Near-Infinite Context
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
• ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
• Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirect
• PipeDream: Fast and Efficient Pipeline Parallel DNN Training
• Horovod: fast and easy distributed deep learning in TensorFlow
• Ray: A Distributed Framework for Emerging AI Applications
• Proximal Policy Optimization Algorithms
• RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS

Cited by

• AReaL-Hex: Accommodating Asynchronous RL Training over Heterogeneous GPUs

Related

• WeChat-YATT: A Scalable, Simple, Efficient, and Production Ready Training Library
• An Adaptive Placement and Parallelism Framework for Accelerating RLHF Training
• RLHFless: Serverless Computing for Efficient RLHF
• Optimizing RLHF Training for Large Language Models with Stage Fusion
• ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation