DeFTX: Denoised Sparse Fine-Tuning for Zero-Shot Cross-Lingual Transfer

2025/05/21 by Sona Elza Simon, Simon, Sona Elza, Preethi Jyothi +1 · 1 citation
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #I.2.7 #Machine Learning (cs.LG) #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2505.15090

openalex publication_date 2025/05/21 · openalex created_date 2025/10/19 · openalex updated_date 2026/07/28

Abstract

Effective cross-lingual transfer remains a critical challenge in scaling the benefits of large language models from high-resource to low-resource languages. Towards this goal, prior studies have explored many approaches to combine task knowledge from task-specific data in a (high-resource) source language and language knowledge from unlabeled text in a (low-resource) target language. One notable approach proposed composable sparse fine-tuning (SFT) for cross-lingual transfer that learns task-specific and language-specific sparse masks to select a subset of the pretrained model's parameters that are further fine-tuned. These sparse fine-tuned vectors (SFTs) are subsequently composed with the pretrained model to facilitate zero-shot cross-lingual transfer to a task in a target language, using only task-specific data from a source language. These sparse masks for SFTs were identified using a simple magnitude-based pruning. In our work, we introduce DeFT-X, a novel composable SFT approach that denoises the weight matrices of a pretrained model before magnitude pruning using singular value decomposition, thus yielding more robust SFTs. We evaluate DeFT-X on a diverse set of extremely low-resource languages for sentiment classification (NusaX) and natural language inference (AmericasNLI) and demonstrate that it performs at par or outperforms SFT and other prominent cross-lingual transfer baselines.

Citations

• Low-Rank and Sparse Model Merging for Multi-Lingual Speech Recognition and Translation
• Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMs
• Encoder vs Decoder: Comparative Analysis of Encoder and Decoder Language Models on Multilingual NLU Tasks
• Multilingual E5 Text Embeddings: A Technical Report
• The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction
• Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch
• TIES-Merging: Resolving Interference When Merging Models
• Distilling Efficient Language-Specific Models for Cross-Lingual Transfer
• NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languages
• The Geometry of Multilingual Language Model Representations
• Composable Sparse Fine-Tuning for Cross-Lingual Transfer
• BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
• UNKs Everywhere: Adapting Multilingual Language Models to New Scripts
• Intrinsic Dimensionality Explains the Effectiveness of Language Model\n Fine-Tuning
• Parameter-Efficient Transfer Learning with Diff Pruning
• IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding
• MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer
• Proving the Lottery Ticket Hypothesis: Pruning is All You Need
• XNLI: Evaluating Cross-lingual Sentence Representations
• The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
• A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
• Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruning
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Learning both Weights and Connections for Efficient Neural Networks

Cited by

• Cross-Prompt Encoder for Low-Performing Languages

Related

• Composable Sparse Fine-Tuning for Cross-Lingual Transfer
• Inducing Generalization across Languages and Tasks using Featurized Low-Rank Mixtures
• Soft Layer Selection with Meta-Learning for Zero-Shot Cross-Lingual Transfer
• One For All & All For One: Bypassing Hyperparameter Tuning with Model Averaging For Cross-Lingual Transfer
• Cross-Lingual Transfer of Debiasing and Detoxification in Multilingual LLMs: An Extensive Investigation