Statement-Tuning Enables Efficient Cross-lingual Generalization in Encoder-only Models

2025/06/02 by Ahmed Elshabrawy, Elshabrawy, Ahmed, Thuvan Nguyen +17
Computer Science · Physics and Astronomy · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Fuzzy Logic and Control Systems #Model Reduction and Neural Networks #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2506.01592

openalex publication_date 2025/06/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) excel in zero-shot and few-shot tasks, but achieving similar performance with encoder-only models like BERT and RoBERTa has been challenging due to their architecture. However, encoders offer advantages such as lower computational and memory costs. Recent work adapts them for zero-shot generalization using Statement Tuning, which reformulates tasks into finite templates. We extend this approach to multilingual NLP, exploring whether encoders can achieve zero-shot cross-lingual generalization and serve as efficient alternatives to memory-intensive LLMs for low-resource languages. Our results show that state-of-the-art encoder models generalize well across languages, rivaling multilingual LLMs while being more efficient. We also analyze multilingual Statement Tuning dataset design, efficiency gains, and language-specific generalization, contributing to more inclusive and resource-efficient NLP models. We release our code and models.

Citations

• The Llama 3 Herd of Models
• Gemma 2: Improving Open Language Models at a Practical Size
• Enabling Natural Zero-Shot Prompting on Encoder Models via Statement-Tuning
• ToPro: Token-Level Prompt Decomposition for Cross-Lingual Sequence Labeling Tasks
• SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialects
• The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants
• Is Prompt-Based Finetuning Always Better than Vanilla Finetuning? Insights from Cross-Lingual Language Understanding
• Recipes for Sequential Pre-training of Multilingual Encoder and Seq2Seq Models
• Multi-lingual and Multi-cultural Figurative Language Understanding
• QLoRA: Efficient Finetuning of Quantized LLMs
• XLM-V: Overcoming the Vocabulary Bottleneck in Multilingual Masked Language Models
• Prompting Large Language Model for Machine Translation: A Case Study
• Self-Instruct: Aligning Language Models with Self-Generated Instructions
• Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor
• A Universal Discriminator for Zero-Shot Generalization
• Prompt-Tuning Can Be Much Better Than Fine-Tuning on Cross-lingual Understanding With Multilingual Language Models
• Scaling Instruction-Finetuned Language Models
• No Language Left Behind: Scaling Human-Centered Machine Translation
• MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages
• ZeroGen: Efficient Zero-shot Learning via Dataset Generation
• Few-shot Learning with Multilingual Language Models
• Efficient Large Scale Language Modeling with Mixtures of Experts
• Discrete and Soft Prompting for Multilingual Models
• Finetuned Language Models Are Zero-Shot Learners
• It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoning
• Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning
• The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation
• Cross-Task Generalization via Natural Language Crowdsourcing Instructions
• Making Pre-trained Language Models Better Few-shot Learners
• EXAMS: A Multi-Subject High School Examinations Dataset for Cross-Lingual and Multilingual Question Answering
• XL-WiC: A Multilingual Benchmark for Evaluating Semantic Contextualization
• WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarization
• The Multilingual Amazon Reviews Corpus
• IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understanding
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• Language Models are Few-Shot Learners
• GoEmotions: A Dataset of Fine-Grained Emotions
• Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference
• Unsupervised Cross-lingual Representation Learning at Scale
• CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data
• On the Cross-lingual Transferability of Monolingual Representations
• Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach
• PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identification
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• XNLI: Evaluating Cross-lingual Sentence Representations
• Character-level Convolutional Networks for Text Classification
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• Multilingual Encoder Knows more than You Realize: Shared Weights Pretraining for Extremely Low-Resource Languages
• Multilingual Sentence-T5: Scalable Sentence Encoders for Multilingual Applications
• An Efficient Multilingual Language Model Compression through Vocabulary Trimming
• An Empirical Study of Factors Affecting Language-Independent Models
• Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarization