Influence Patterns for Explaining Information Flow in BERT

2020/11/02 by Kaiji Lu, Zifan Wang, Lu, Kaiji +5 · 2 citations
Computer Science · #Computation and Language (cs.CL) #Data Stream Mining Techniques #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2011.00740

openalex publication_date 2020/11/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

While attention is all you need may be proving true, we do not know why: attention-based transformer models such as BERT are superior but how information flows from input tokens to output predictions are unclear. We introduce influence patterns, abstractions of sets of paths through a transformer model. Patterns quantify and localize the flow of information to paths passing through a sequence of model nodes. Experimentally, we find that significant portion of information flow in BERT goes through skip connections instead of attention heads. We further show that consistency of patterns across instances is an indicator of BERT's performance. Finally, We demonstrate that patterns account for far more model performance than previous attention-based and layer-based methods.

Citations

• Axiomatic Attribution for Deep Networks
• SmoothGrad: removing noise by adding noise
• LLM-Supported Manufacturing Mapping Generation
• When BERT Plays the Lottery, All Tickets Are Winning
• Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depth

Cited by

• Cross-Modal Unlearning via Influential Neuron Path Editing in Multimodal Large Language Models
• SafeGuider: Robust and Practical Content Safety Control for Text-to-Image Models

Related

• Telling BERT's full story: from Local Attention to Global Aggregation
• Breaking BERT: Evaluating and Optimizing Sparsified Attention
• Self-Attention Attribution: Interpreting Information Interactions Inside Transformer
• Revealing the Dark Secrets of BERT
• How Far Does BERT Look At:Distance-based Clustering and Analysis of BERT's Attention