Learning to Route Languages for Multilingual Policy Optimization

2026/05/25 by Geyang Guo, Hiromi Wakaki, Yuki Mitsufuji +2 · 1 voice
Computer Science · #Constructed language #Diversity (politics) #Language model #Multilingualism #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Quality (philosophy) #Reinforcement learning #Router #Topic Modeling #cs.CL

paper · pdf · open access · doi:10.48550/arxiv.2605.25360

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2026/05/25 · arxiv published 2026/05/25 · arxiv updated 2026/05/25 · openalex created_date 2026/05/27 · openalex updated_date 2026/07/28

Abstract

Large language models~(LLMs) are trained on heterogeneous multilingual corpora, yet existing policy optimization methods often implicitly restrict each training question to a single response language or rely on a fixed dominant language for supervision. We propose language-routed policy optimization (LRPO), an online policy optimization framework that treats language as a selectable variable. LRPO elicits multilingual rollouts for each training question and integrates their relative quality into preference-based policy updates, increasing the diversity and informativeness of training signals under the fixed rollout budget. To adaptively determine which languages to explore during reinforcement learning, we introduce a trainable language router formulated as a multi-armed bandit, balancing exploration of underutilized languages with exploitation of more informative ones. Extensive experiments show that LRPO consistently improves multilingual performance, demonstrating that adaptive language routing enables effective cross-lingual knowledge exploitation for training. We release all the resources at https://github.com/Guochry/LRPO.

Citations

• Early science acceleration experiments with GPT-5
• Mind the Gap... or Not? How Translation Errors and Evaluation Details Skew Multilingual Results
• Think Natively: Unlocking Multilingual Reasoning with Consistency-Enhanced Reinforcement Learning
• Pushing on Multilingual Reasoning Models with Language-Mixed Chain-of-Thought
• mmBERT: A Modern Multilingual Encoder with Annealed Language Learning
• The Impact of Language Mixing on Bilingual LLM Reasoning
• Are Knowledge and Reference in Multilingual Language Models Cross-Lingually Consistent?
• CCL-XCoT: An Efficient Cross-Lingual Knowledge Transfer Method for Mitigating Hallucination Generation
• Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Dataset
• Learn Globally, Speak Locally: Bridging the Gaps in Multilingual Reasoning
• MPO: Multilingual Safety Alignment via Reward Gap Optimization
• When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual Reasoners
• BLEUBERI: BLEU is a surprisingly effective reward for instruction following
• HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages
• Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?
• Can you map it to English? The Role of Cross-Lingual Alignment in Multilingual Performance of LLMs
• CARE: Multilingual Human Preference Learning for Cultural Awareness
• Lost in Multilinguality: Dissecting Cross-lingual Factual Inconsistency in Transformer Language Models
• Gemma 3 Technical Report
• Qwen2.5 Technical Report
• Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation
• INCLUDE: Evaluating Multilingual Language Understanding with Regional Knowledge
• Measuring short-form factuality in large language models
• Language Imbalance Driven Rewarding for Multilingual Self-improving
• CulturalBench: A Robust, Diverse, and Challenging Cultural Benchmark by Human-AI CulturalTeaming
• The Llama 3 Herd of Models
• NativQA: Multilingual Culturally-Aligned Natural Query for LLMs
• 1+1>2: Can Large Language Models Serve as Cross-Lingual Knowledge Aggregators?
• LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models
• Is Translation All You Need? A Study on Solving Multilingual Tasks with Large Language Models
• Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning
• BioXP-0.5B: Explainable Medical-AI via RL-GRPO
• MAPO: Advancing Multilingual Reasoning through Multilingual Alignment-as-Preference Optimization
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Training language models to follow instructions with human feedback
• Training Verifiers to Solve Math Word Problems

Discussions

• Might be relevant: https://arxiv.org/abs/2605.25360 — Uriel Dolev [bsky, 0 points, 0 comments]

Related

• Meta-Learning Preferences for Multilingual LLM Alignment
• Bootstrapping LLMs via Preference-Based Policy Optimization
• MAPO: Advancing Multilingual Reasoning through Multilingual Alignment-as-Preference Optimization
• RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs
• Multi-Objective Reinforcement Learning for Large Language Model Optimization: Visionary Perspective