How Does Alignment Enhance LLMs' Multilingual Capabilities? A Language Neurons Perspective

2025/05/27 by Shimao Zhang, Zhang, Shimao, Zhejian Lai +13
Arts and Humanities · Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Text Readability and Simplification #Translation Studies and Practices

paper · pdf · doi:10.48550/arxiv.2505.21505

openalex publication_date 2025/05/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Multilingual Alignment is an effective and representative paradigm to enhance LLMs' multilingual capabilities, which transfers the capabilities from the high-resource languages to the low-resource languages. Meanwhile, some research on language-specific neurons provides a new perspective to analyze and understand LLMs' mechanisms. However, we find that there are many neurons that are shared by multiple but not all languages and cannot be correctly classified. In this work, we propose a ternary classification methodology that categorizes neurons into three types, including language-specific neurons, language-related neurons, and general neurons. And we propose a corresponding identification algorithm to distinguish these different types of neurons. Furthermore, based on the distributional characteristics of different types of neurons, we divide the LLMs' internal process for multilingual inference into four parts: (1) multilingual understanding, (2) shared semantic space reasoning, (3) multilingual output space transformation, and (4) vocabulary space outputting. Additionally, we systematically analyze the models before and after alignment with a focus on different types of neurons. We also analyze the phenomenon of "Spontaneous Multilingual Alignment". Overall, our work conducts a comprehensive investigation based on different types of neurons, providing empirical results and valuable insights to better understand multilingual alignment and multilingual capabilities of LLMs.

Citations

• Process-based Self-Rewarding Language Models
• SLAM: Towards Efficient Multilingual Reasoning via Selective Language Alignment
• Qwen2.5 Technical Report
• EMMA-500: Enhancing Massively Multilingual Adaptation of Large Language Models
• The Llama 3 Herd of Models
• Large Language Models Are Cross-Lingual Knowledge-Free Reasoners
• Exploring Alignment in Shared Cross-lingual Spaces
• Getting More from Less: Large Language Models are Good Spontaneous Multilingual Learners
• Neuron Specialization: Leveraging intrinsic task modularity for multilingual machine translation
• On the Multilingual Ability of Decoder-based Pre-trained Language Models: Finding and Controlling Language-Specific Neurons
• AdaMergeX: Cross-Lingual Transfer with Large Language Models via Adaptive Adapter Merging
• How do Large Language Models Handle Multilingualism?
• Language-Specific Neurons: The Key to Multilingual Capabilities in Large Language Models
• Do Llamas Work in English? On the Latent Language of Multilingual Transformers
• Question Translation Training for Better Multilingual Reasoning
• MAPO: Advancing Multilingual Reasoning through Multilingual Alignment-as-Preference Optimization
• LLaMA Beyond English: An Empirical Study on Language Capability Transfer
• Breaking Language Barriers in Multilingual Mathematical Reasoning: Insights and Observations
• MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Direct Preference Optimization: Your Language Model is Secretly a Reward Model
• Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting
• Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
• Language Models are Multilingual Chain-of-Thought Reasoners
• NumGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks
• Continual Mixed-Language Pre-Training for Extremely Low-Resource Neural Machine Translation
• GLU Variants Improve Transformer
• Proximal Policy Optimization Algorithms

Related

• From Neurons to Semantics: Evaluating Cross-Linguistic Alignment Capabilities of Large Language Models via Neurons Alignment
• Converging to a Lingua Franca: Evolution of Linguistic Regions and Semantics Alignment in Multilingual Large Language Models
• Language Arithmetics: Towards Systematic Language Neuron Identification and Manipulation
• Sharing Matters: Analysing Neurons Across Languages and Tasks in LLMs
• Probing the Emergence of Cross-lingual Alignment during LLM Training