Group-Aware Reinforcement Learning for Output Diversity in Large Language Models

2025/11/16 by Oron Anschel, Anschel, Oron, Alon Shoshan +17 · 1 citation
Computer Science · Medicine · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Healthcare and Education #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2511.12596

openalex publication_date 2025/11/16 · openalex created_date 2025/11/19 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) often suffer from mode collapse, repeatedly generating the same few completions even when many valid answers exist, limiting their diversity across a wide range of tasks. We introduce Group-Aware Policy Optimization (GAPO), a simple extension of the recent and popular Group Relative Policy Optimization (GRPO) that computes rewards over the group as a whole. GAPO enables learning from the group-level properties such as diversity and coverage. We demonstrate GAPO using a frequency-aware reward function that encourages uniform sampling over valid LLM completions, and show that GAPO-trained models produce valid and more diverse model responses. Beyond this setup, GAPO generalizes to open-ended prompts and improves response diversity without compromising accuracy on standard LLM benchmarks (GSM8K, MATH, HumanEval, MMLU-Pro). Our code will be made publicly available.

Citations

• Diverse Preference Learning for Capabilities and Alignment
• SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training
• Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs
• MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• Reducing Selection Bias in Large Language Models
• Generative Flow Networks as Entropy-Regularized RL
• Understanding the Effects of RLHF on LLM Generalisation and Diversity
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Training language models to follow instructions with human feedback
• Training Verifiers to Solve Math Word Problems
• Evaluating Large Language Models Trained on Code
• Measuring Mathematical Problem Solving With the MATH Dataset
• MPNet: Masked and Permuted Pre-training for Language Understanding
• Fine-Tuning Language Models from Human Preferences
• Neural Text Generation with Unlikelihood Training
• The Curious Case of Neural Text Degeneration
• Hierarchical Neural Story Generation
• Texygen: A Benchmarking Platform for Text Generation Models
• Proximal Policy Optimization Algorithms
• Trust Region Policy Optimization

Cited by

• CIEGAD: Cluster-Conditioned Interpolative and Extrapolative Framework for Geometry-Aware and Domain-Aligned Data Augmentation

Related

• DRA-GRPO: Exploring Diversity-Aware Reward Adjustment for R1-Zero-Like Training of Large Language Models
• GRPO-RM: Fine-Tuning Representation Models via GRPO-Driven Reinforcement Learning
• Group Sequence Policy Optimization
• GVPO: Group Variance Policy Optimization for Large Language Model Post-Training
• Stepwise Guided Policy Optimization: Coloring your Incorrect Reasoning in GRPO