Analyzing Individual Neurons in Pre-trained Language Models

2020/10/06 by Nadir Durrani, Hassan Sajjad, Durrani, Nadir +5 · 10 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #cs.CL

paper · pdf · doi:10.48550/arxiv.2010.02695

Accepted in EMNLP 2020

arxiv created 2020/10/06 · arxiv updated 2020/10/07

Abstract

While a lot of analysis has been carried to demonstrate linguistic knowledge captured by the representations learned within deep NLP models, very little attention has been paid towards individual neurons.We carry outa neuron-level analysis using core linguistic tasks of predicting morphology, syntax and semantics, on pre-trained language models, with questions like: i) do individual neurons in pre-trained models capture linguistic information? ii) which parts of the network learn more about certain linguistic phenomena? iii) how distributed or focused is the information? and iv) how do various architectures differ in learning these properties? We found small subsets of neurons to predict linguistic tasks, with lower level tasks (such as morphology) localized in fewer neurons, compared to higher level task of predicting syntax. Our study also reveals interesting cross architectural comparisons. For example, we found neurons in XLNet to be more localized and disjoint when predicting properties compared to BERT and others, where they are more distributed and coupled.

Cited by

• ParaUni: Enhance Generation in Unified Multimodal Model with Reinforcement-driven Hierarchical Parallel Information Interaction
• Generating an Image From 1,000 Words: Enhancing Text-to-Image With Structured Captions
• On the Analogy between Human Brain and LLMs: Spotting Key Neurons in Grammar Perception
• UniFusion: Vision-Language Model as Unified Encoder in Image Generation
• Continual Personalization for Diffusion Models
• RepreGuard: Detecting LLM-Generated Text by Revealing Hidden Representation Patterns
• Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules
• Distilling Knowledge from Large Language Models: A Concept Bottleneck Model for Hate and Counter Speech Recognition
• On the Performance of Concept Probing: The Influence of the Data (Extended Version)
• Toward the Explainability of Protein Language Models
• Linguistic Interpretability of Transformer-based Language Models: a systematic review

Related

• Discovering Salient Neurons in Deep NLP Models
• What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models
• How transfer learning impacts linguistic knowledge in deep NLP models?
• Finding Neurons in a Haystack: Case Studies with Sparse Probing
• On the Pitfalls of Analyzing Individual Neurons in Language Models