Deriving Machine Attention from Human Rationales

2018/08/28 by Yujia Bao, Shiyu Chang, Bao, Yujia +5 · 10 citations
Computer Science · #Computation and Language (cs.CL) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1808.09367

openalex publication_date 2018/08/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Attention-based models are successful when trained on large amounts of data. In this paper, we demonstrate that even in the low-resource scenario, attention can be learned effectively. To this end, we start with discrete human-annotated rationales and map them into continuous attention. Our central hypothesis is that this mapping is general across domains, and thus can be transferred from resource-rich domains to low-resource ones. Our model jointly learns a domain-invariant representation and induces the desired mapping between rationales and attention. Our empirical results validate this hypothesis and show that our approach delivers significant gains over state-of-the-art baselines, yielding over 15% average error reduction on benchmark datasets.

Citations

• Categorical Reparameterization with Gumbel-Softmax
• Improved Training of Wasserstein GANs
• Convolutional Neural Networks for Sentence Classification

Cited by

• From Fake Focus to Real Precision: Confusion-Driven Adversarial Attention Learning in Transformers
• Text Rationalization for Robust Causal Effect Estimation
• Learnable Game-theoretic Policy Optimization for Data-centric Self-explanation Rationalization
• Interpretations are useful: penalizing explanations to align neural networks with prior knowledge
• Can human clinical rationales improve the performance and explainability of clinical text classification models?
• Text Classification with Lexicon from PreAttention Mechanism
• Evaluating Explanation Without Ground Truth in Interpretable Machine\n Learning
• Machine Learning from Explanations
• SFT-GO: Supervised Fine-Tuning with Group Optimization for Large Language Models
• Efficient Text-Attributed Graph Learning through Selective Annotation and Graph Alignment

Related

• Cost-effective Interactive Attention Learning with Neural Attention Processes
• Learning to Look: Cognitive Attention Alignment with Vision-Language Models
• Aligning Human and Machine Attention for Enhanced Supervised Learning
• Attention: Marginal Probability is All You Need?
• Unified Attention Modeling for Efficient Free-Viewing and Visual Search via Shared Representations