Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback

2023/07/29 by Viet Dac Lai, Chien Van Nguyen, Lai, Viet Dac +11 · 27 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Speech and dialogue systems #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2307.16039

openalex publication_date 2023/07/29 · openalex created_date 2023/08/02 · openalex updated_date 2026/07/28

Abstract

A key technology for the development of large language models (LLMs) involves instruction tuning that helps align the models' responses with human expectations to realize impressive learning abilities. Two major approaches for instruction tuning characterize supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF), which are currently applied to produce the best commercial LLMs (e.g., ChatGPT). To improve the accessibility of LLMs for research and development efforts, various instruction-tuned open-source LLMs have also been introduced recently, e.g., Alpaca, Vicuna, to name a few. However, existing open-source LLMs have only been instruction-tuned for English and a few popular languages, thus hindering their impacts and accessibility to many other languages in the world. Among a few very recent work to explore instruction tuning for LLMs in multiple languages, SFT has been used as the only approach to instruction-tune LLMs for multiple languages. This has left a significant gap for fine-tuned LLMs based on RLHF in diverse languages and raised important questions on how RLHF can boost the performance of multilingual instruction tuning. To overcome this issue, we present Okapi, the first system with instruction-tuned LLMs based on RLHF for multiple languages. Okapi introduces instruction and response-ranked data in 26 diverse languages to facilitate the experiments and development of future multilingual LLM research. We also present benchmark datasets to enable the evaluation of generative LLMs in multiple languages. Our experiments demonstrate the advantages of RLHF for multilingual instruction over SFT for different base models and datasets. Our framework and resources are released at https://github.com/nlp-uoregon/Okapi.

Cited by

• Gamayun's Path to Multilingual Mastery: Cost-Efficient Training of a 1.5B-Parameter LLM
• qa-FLoRA: Data-free query-adaptive Fusion of LoRAs for LLMs
• Fluent Alignment with Disfluent Judges: Post-training for Lower-resource Languages
• Elastic Mixture of Rank-Wise Experts for Knowledge Reuse in Federated Fine-Tuning
• Estonian WinoGrande Dataset: Comparative Analysis of LLM Performance on Human and Machine Translation
• Donors and Recipients: On Asymmetric Transfer Across Tasks and Languages with Parameter-Efficient Fine-Tuning
• MTQ-Eval: Multilingual Text Quality Evaluation for Language Models
• CAPO: Confidence Aware Preference Optimization Learning for Multilingual Preferences
• LiveCLKTBench: Towards Reliable Evaluation of Cross-Lingual Knowledge Transfer in Multilingual LLMs
• Do You Know About My Nation? Investigating Multilingual Language Models' Cultural Literacy Through Factual Knowledge
• Rethinking Cross-lingual Alignment: Balancing Transfer and Cultural Erasure in Multilingual LLMs
• Revisiting Multilingual Data Mixtures in Language Model Pretraining
• Mitigating Cross-Lingual Cultural Inconsistencies in LLMs via Consensus-Driven Preference Optimisation
• Preserving LLM Capabilities through Calibration Data Curation: From Analysis to Optimization
• AlignX: Advancing Multilingual Large Language Models with Multilingual Representation Alignment
• CM-Align: Consistency-based Multilingual Alignment for Large Language Models
• Llama-GENBA-10B: A Trilingual Large Language Model for German, English and Bavarian
• Benchmarking Hindi LLMs: A New Suite of Datasets and a Comparative Analysis
• TASE: Token Awareness and Structured Evaluation for Multilingual Language Models
• Whose Truth? Pluralistic Geo-Alignment for (Agentic) AI
• Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization
• RegMean++: Enhancing Effectiveness and Generalization of Regression Mean for Model Merging
• IndoPref: A Multi-Domain Pairwise Preference Dataset for Indonesian
• Uncovering Cross-Linguistic Disparities in LLMs using Sparse Autoencoders
• CodeMixBench: Evaluating Code-Mixing Capabilities of LLMs Across 18 Languages
• From Neurons to Semantics: Evaluating Cross-Linguistic Alignment Capabilities of Large Language Models via Neurons Alignment
• The AI Language Proficiency Monitor -- Tracking the Progress of LLMs on Multilingual Benchmarks

Related

• Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpaca
• Building Instruction-Tuning Datasets from Human-Written Instructions with Open-Weight Large Language Models
• Investigating Multilingual Instruction-Tuning: Do Polyglot Models Demand for Multilingual Instructions?
• Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptation
• LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback