It's All in the Heads: Using Attention Heads as a Baseline for Cross-Lingual Transfer in Commonsense Reasoning

2021/06/22 by Alexey Tikhonov, Max Ryabinin, Tikhonov, Alexey +1 · 13 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2106.12066

Accepted to Findings of ACL 2021. 13 pages, 4 figures. Code: https://github.com/yandex-research/crosslingual_winograd

openalex publication_date 2021/06/22 · arxiv created 2021/11/30 · arxiv updated 2021/12/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Commonsense reasoning is one of the key problems in natural language processing, but the relative scarcity of labeled data holds back the progress for languages other than English. Pretrained cross-lingual models are a source of powerful language-agnostic representations, yet their inherent reasoning capabilities are still actively studied. In this work, we design a simple approach to commonsense reasoning which trains a linear classifier with weights of multi-head attention as features. To evaluate this approach, we create a multilingual Winograd Schema corpus by processing several datasets from prior work within a standardized pipeline and measure cross-lingual generalization ability in terms of out-of-sample performance. The method performs competitively with recent supervised and unsupervised approaches for commonsense reasoning, even when applied to other languages in a zero-shot manner. Also, we demonstrate that most of the performance is given by the same small subset of attention heads for all studied languages, which provides evidence of universal reasoning capabilities in multilingual encoders.

Cited by

• Donors and Recipients: On Asymmetric Transfer Across Tasks and Languages with Parameter-Efficient Fine-Tuning
• Revisiting Multilingual Data Mixtures in Language Model Pretraining
• MeTA-LoRA: Data-Efficient Multi-Task Fine-Tuning for Large Language Models
• Parity-Aware Byte-Pair Encoding: Improving Cross-lingual Fairness in Tokenization
• Learn Globally, Speak Locally: Bridging the Gaps in Multilingual Reasoning
• MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretraining
• Graph-of-Causal Evolution: Challenging Chain-of-Model for Reasoning
• Just Go Parallel: Improving the Multilingual Capabilities of Large Language Models
• Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning
• TokAlign: Efficient Vocabulary Adaptation via Token Alignment
• Statement-Tuning Enables Efficient Cross-lingual Generalization in Encoder-only Models
• Rewriting Pre-Training Data Boosts LLM Performance in Math and Code
• Structured Recurrent Mixers for Massively Parallelized Sequence Generation

Related

• Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning
• CLICKER: Attention-Based Cross-Lingual Commonsense Knowledge Transfer
• Leveraging Knowledge in Multilingual Commonsense Reasoning
• MetaXCR: Reinforcement-Based Meta-Transfer Learning for Cross-Lingual Commonsense Reasoning
• Attention Is (not) All You Need for Commonsense Reasoning