Fluent Alignment with Disfluent Judges: Post-training for Lower-resource Languages

2025/12/09 by Samuel, David, Øvrelid, Lilja, Velldal, Erik +1
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling

paper · doi:10.48550/arxiv.2512.08777

openalex publication_date 2025/12/09 · openalex created_date 2025/12/11 · openalex updated_date 2026/07/28

Abstract

We propose a post-training method for lower-resource languages that preserves the fluency of language models even when aligned by disfluent reward models. Preference optimization is now a well-researched topic, but previous work has mostly addressed models for English and Chinese. Lower-resource languages lack both datasets written by native speakers and instruction-tuned language models capable of generating fluent synthetic data. To address this, we focus on developing a fluent preference-aligned language model without any instruction-tuning data in the target language. Our approach uses an on-policy training method, which we compare with two common alternatives: supervised finetuning on machine-translated data and multilingual finetuning. We conduct a case study on Norwegian Bokmål and evaluate fluency through native-speaker assessments. The results show that the on-policy aspect is crucial and outperforms the alternatives without relying on any hard-to-obtain data.

Citations

• Expanding Foundational Language Capabilities in Open-Source LLMs through a Korean Case Study
• Seed-X: Building Strong Multilingual Translation LLM with 7B Parameters
• Instructing Large Language Models for Low-Resource Languages: A Systematic Study for Basque
• NorEval: A Norwegian Language Understanding and Generation Evaluation Benchmark
• LinguaLIFT: An Effective Two-stage Instruction Tuning Framework for Low-Resource Language Reasoning
• The Impact of Copyrighted Material on Large Language Models: A Norwegian Perspective
• Do Large Language Models Have an English Accent? Evaluating and Improving the Naturalness of Multilingual LLMs
• The Llama 3 Herd of Models
• ALLaM: Large Language Models for Arabic and English
• RLHF Can Speak Many Languages: Unlocking Multilingual Preference Optimization for LLMs
• Quantifying the Gain in Weak-to-Strong Generalization
• Simple and Scalable Strategies to Continually Pre-train Large Language Models
• Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model
• Self-Rewarding Language Models
• Typhoon: Thai Large Language Models
• Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision
• SeaLLMs -- Large Language Models for Southeast Asia
• MADLAD-400: A Multilingual And Document-Level Large Audited Dataset
• From Base to Conversational: Japanese Instruction Dataset and Tuning Large Language Models
• Camoscio: an Italian Instruction-tuned LLaMA
• Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• LIMA: Less Is More for Alignment
• NorQuAD: Norwegian Question Answering Dataset
• Stable and low-precision training for large-scale vision-language models
• Understanding Translationese in Cross-Lingual Summarization
• Democratizing Neural Machine Translation with OPUS-MT
• No Language Left Behind: Scaling Human-Centered Machine Translation
• Training language models to follow instructions with human feedback
• mT5: A massively multilingual pre-trained text-to-text transformer
• Learning to summarize from human feedback
• Don't Stop Pretraining: Adapt Language Models to Domains and Tasks
• Unsupervised Cross-lingual Representation Learning at Scale
• Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog
• Lost in Translation: Loss and Decay of Linguistic Richness in Machine Translation
• How multilingual is Multilingual BERT?
• The Curious Case of Neural Text Degeneration
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Unsupervised Machine Translation Using Monolingual Corpora Only
• Unsupervised Neural Machine Translation
• NoReC: The Norwegian Review Corpus
• Proximal Policy Optimization Algorithms
• Deep reinforcement learning from human preferences
• The Optimal Reward Baseline for Gradient-Based Reinforcement Learning

Related

• A method for improving multilingual quality and diversity of instruction fine-tuning datasets
• CM-Align: Consistency-based Multilingual Alignment for Large Language Models
• MultiFiT: Efficient Multi-lingual Language Model Fine-tuning
• LIONs: An Empirically Optimized Approach to Align Language Models
• Smaller Language Models are capable of selecting Instruction-Tuning Training Data for Larger Language Models