Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignment

2023/11/14 by Chong Li, Li, Chong, Shaonan Wang +5 · 8 citations
Computer Science · Social Sciences · #Computation and Language (cs.CL) #Computational and Text Analysis Methods #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2311.08089

openalex publication_date 2023/11/14 · openalex created_date 2023/11/16 · openalex updated_date 2026/07/28

Abstract

Multilingual generative models obtain remarkable cross-lingual in-context learning capabilities through pre-training on large-scale corpora. However, they still exhibit a performance bias toward high-resource languages and learn isolated distributions of multilingual sentence representations, which may hinder knowledge transfer across languages. To bridge this gap, we propose a simple yet effective cross-lingual alignment framework exploiting pairs of translation sentences. It aligns the internal sentence representations across different languages via multilingual contrastive learning and aligns outputs by following cross-lingual instructions in the target language. Experimental results show that even with less than 0.1 \textperthousand of pre-training tokens, our alignment framework significantly boosts the cross-lingual abilities of generative language models and mitigates the performance gap. Further analyses reveal that it results in a better internal multilingual representation distribution of multilingual models.

Cited by

• Unlocking Multilingual Reasoning Capability of LLMs and LVLMs through Representation Engineering
• Code-Switching In-Context Learning for Cross-Lingual Transfer of Large Language Models
• SoT: Structured-of-Thought Prompting Guides Multilingual Reasoning in Large Language Models
• AlignX: Advancing Multilingual Large Language Models with Multilingual Representation Alignment
• A Survey of Large Language Models in Discipline-specific Research: Challenges, Methods and Opportunities
• TokAlign: Efficient Vocabulary Adaptation via Token Alignment
• When Language Shapes Thought: Cross-Lingual Transfer of Factual Knowledge in Question Answering
• Hierarchical Level-Wise News Article Clustering via Multilingual Matryoshka Embeddings
• MPO: Multilingual Safety Alignment via Reward Gap Optimization

Related

• Dual-Alignment Pre-training for Cross-lingual Sentence Embedding
• Align, Generate, Learn: A Novel Closed-Loop Framework for Cross-Lingual In-Context Learning
• Cross-lingual Alignment vs Joint Training: A Comparative Study and A Simple Unified Framework
• Cross-Align: Modeling Deep Cross-lingual Interactions for Word Alignment
• VECO 2.0: Cross-lingual Language Model Pre-training with Multi-granularity Contrastive Learning