DEMix Layers: Disentangling Domains for Modular Language Modeling

2021/08/11 by Suchin Gururangan, Gururangan, Suchin, Lewis, Mike +6 · 12 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2108.05036

openalex publication_date 2021/08/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce a new domain expert mixture (DEMix) layer that enables conditioning a language model (LM) on the domain of the input text. A DEMix layer is a collection of expert feedforward networks, each specialized to a domain, that makes the LM modular: experts can be mixed, added or removed after initial training. Extensive experiments with autoregressive transformer LMs (up to 1.3B parameters) show that DEMix layers reduce test-time perplexity, increase training efficiency, and enable rapid adaptation with little overhead. We show that mixing experts during inference, using a parameter-free weighted ensemble, allows the model to better generalize to heterogeneous or unseen domains. We also show that experts can be added to iteratively incorporate new domains without forgetting older ones, and that experts can be removed to restrict access to unwanted domains, without additional training. Overall, these results demonstrate benefits of explicitly conditioning on textual domains during language modeling.

Cited by

• Mitigating Forgetting in Low Rank Adaptation
• Beyond Data Filtering: Knowledge Localization for Capability Removal in LLMs
• CREST: Improving Interpretability and Effectiveness of Troubleshooting at Ericsson through Criterion-Specific Trouble Report Retrieval
• AnaCP: Toward Upper-Bound Continual Learning via Analytic Contrastive Projection
• Robust Uncertainty Quantification for Self-Evolving Large Language Models via Continual Domain Pretraining
• Neuronal Group Communication for Efficient Neural representation
• Analyzing and Internalizing Complex Policy Documents for LLM Agents
• JE-IRT: A Geometric Lens on LLM Abilities through Joint Embedding Item Response Theory
• Opening the Black Box: Interpretable LLMs via Semantic Resonance Architecture
• Mixture-of-Clustered-Experts: Advancing Expert Specialization and Generalization in Instruction Tuning
• ExpertSim: Fast Particle Detector Simulation Using Mixture-of-Generative-Experts
• Revisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Models

Related

• Flexible and Effective Mixing of Large Language Models into a Mixture of Domain Experts
• Scalable Multi-Domain Adaptation of Language Models using Modular Experts
• Efficient Large Scale Language Modeling with Mixtures of Experts
• MoLEx: Mixture of Layer Experts for Finetuning with Sparse Upcycling
• Expert Merging: Model Merging with Unsupervised Expert Alignment and Importance-Guided Layer Chunking