Relic: Enhancing Reward Model Generalization for Low-Resource Indic Languages with Few-Shot Examples

2025/06/19 by Ghosal, Soumya Suvra, Singh, Vaibhav, Ghosh, Akash +4 · 3 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning and Data Classification #Recommender Systems and Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2506.16502

openalex publication_date 2025/06/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Reward models are essential for aligning large language models (LLMs) with human preferences. However, most open-source multilingual reward models are primarily trained on preference datasets in high-resource languages, resulting in unreliable reward signals for low-resource Indic languages. Collecting large-scale, high-quality preference data for these languages is prohibitively expensive, making preference-based training approaches impractical. To address this challenge, we propose RELIC, a novel in-context learning framework for reward modeling in low-resource Indic languages. RELIC trains a retriever with a pairwise ranking objective to select in-context examples from auxiliary high-resource languages that most effectively highlight the distinction between preferred and less-preferred responses. Extensive experiments on three preference datasets- PKU-SafeRLHF, WebGPT, and HH-RLHF-using state-of-the-art open-source reward models demonstrate that RELIC significantly improves reward model accuracy for low-resource Indic languages, consistently outperforming existing example selection methods. For example, on Bodo-a low-resource Indic language-using a LLaMA-3.2-3B reward model, RELIC achieves a 12.81% and 10.13% improvement in accuracy over zero-shot prompting and state-of-the-art example selection method, respectively.

Citations

• Mind the (Language) Gap: Towards Probing Numerical and Cross-Lingual Limits of LVLMs
• A Survey of Multilingual Reasoning in Language Models
• PromptRefine: Enhancing Few-Shot Performance on Low-Resource Indic Languages with Example Selection from Related Example Banks
• Opportunities and Challenges of Large Language Models for Low-Resource Languages in Humanities Research
• Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignment
• Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs
• Cross-lingual Transfer of Reward Models in Multilingual Alignment
• M-RewardBench: Evaluating Reward Models in Multilingual Settings
• The Llama 3 Herd of Models
• PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference
• Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs
• Transfer Q Star: Principled Decoding for LLM Alignment
• XAMPLER: Learning to Retrieve Cross-Lingual In-Context Examples
• IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languages
• Reuse Your Rewards: Reward Model Transfer for Zero-Shot Cross-Lingual Alignment
• ARGS: Alignment as Reward-Guided Search
• MAPO: Advancing Multilingual Reasoning through Multilingual Alignment-as-Preference Optimization
• In-Context Learning with Iterative Demonstration Selection
• Do Multilingual Language Models Think Better in English?
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Dr.ICL: Demonstration-Retrieved In-context Learning
• RetICL: Sequential Retrieval of In-Context Examples with Reinforcement Learning
• Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting
• Multilingual LLMs are Better Cross-lingual In-context Learners with Alignment
• Unified Demonstration Retriever for In-Context Learning
• Finding Support Examples for In-Context Learning
• In-context Example Selection with Influences
• Compositional Exemplars for In-context Learning
• Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Ordering
• Diverse Demonstrations Improve In-context Compositional Generalization
• Active Example Selection for In-Context Learning
• How Do Multilingual Encoders Learn Cross-lingual Representation?
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• WebGPT: Browser-assisted question-answering with human feedback
• Learning To Retrieve Prompts for In-Context Learning
• Language Models are Few-shot Multilingual Learners
• Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners
• What Makes Good In-Context Examples for GPT-3?
• Learning to summarize from human feedback
• Dense Passage Retrieval for Open-Domain Question Answering
• Fine-Tuning Language Models from Human Preferences
• Latent Retrieval for Weakly Supervised Open Domain Question Answering
• Cross-lingual Language Model Pretraining
• Deep reinforcement learning from human preferences
• RANK ANALYSIS OF INCOMPLETE BLOCK DESIGNS
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• CLINIC: Evaluating Multilingual Trustworthiness in Language Models for Healthcare
• Let's Play Across Cultures: A Large Multilingual, Multicultural Benchmark for Assessing Language Models' Understanding of Sports
• KITE: Kernelized and Information Theoretic Exemplars for In-Context Learning

Related

• A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Future
• PromptRefine: Enhancing Few-Shot Performance on Low-Resource Indic Languages with Example Selection from Related Example Banks
• Boosting Reward Model with Preference-Conditional Multi-Aspect Synthetic Data Generation
• RewardBench: Evaluating Reward Models for Language Modeling
• GRAM: A Generative Foundation Reward Model for Reward Generalization