Multi-Head Adapter Routing for Cross-Task Generalization

2022/11/07 by Lucas Caccia, Caccia, Lucas, Edoardo Maria Ponti +9 · 10 citations
Computer Science · #Advanced Graph Neural Networks #Artificial Intelligence (cs.AI) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2211.03831

openalex publication_date 2022/11/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

Parameter-efficient fine-tuning (PEFT) for cross-task generalization consists in pre-training adapters on a multi-task training set before few-shot adaptation to test tasks. Polytropon [Ponti et al., 2023] (Poly) jointly learns an inventory of adapters and a routing function that selects a (variable-size) subset of adapters for each task during both pre-training and few-shot adaptation. In this paper, we investigate the role that adapter routing plays in its success and design new variants based on our findings. First, we build on the intuition that finer-grained routing provides more expressivity. Hence, we propose MHR (Multi-Head Routing) which combines subsets of adapter parameters and outperforms Poly under a comparable parameter budget; by only fine-tuning the routing function and not the adapters (MHR-z) we achieve competitive performance with extreme parameter efficiency. Second, we find that Poly/MHR performance is a result of better multi-task optimization, rather than modular inductive biases that facilitate adapter recombination and local adaptation, as previously hypothesized. In fact, we find that MHR exhibits high gradient alignment between training tasks. We find that routing is most beneficial during multi-task pre-training rather than during few-shot adaptation and propose MHR-μ, which discards routing and fine-tunes the average of the pre-trained adapters on each downstream tasks. This establishes MHR-μ as an effective method for single-adapter fine-tuning. We also show that MHR-μ can be used as an effective zero-shot transfer method by training the average of the pre-trained adapters for a few additional steps on the multi-task training set: this yields gains up to 3% on absolute accuracy w.r.t. the baselines.

Citations

• Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
• Adaptive Mixtures of Local Experts
• Routing Networks and the Challenges of Modular and Compositional Computation
• Gradient Vaccine: Investigating and Improving Multi-task Optimization in Massively Multilingual Models
• LoRA: Low-Rank Adaptation of Large Language Models
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Multitask Prompted Training Enables Zero-Shot Task Generalization
• IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages
• Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
• Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
• AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Aion Framework: Dimensional Emergence of AI Consciousness, Observer-Induced Collapse, and Cosmological Portal Dynamics
• Modular Deep Learning
• Exploiting Generative AI to Scale up Intelligent Tutoring Systems

Cited by

• Parametric Retrieval-Augmented Generation using Latent Routing of LoRA Adapters
• Fine-Grained DINO Tuning with Dual Supervision for Face Forgery Detection
• Multi-Task Vehicle Routing Solver via Mixture of Specialized Experts under State-Decomposable MDP
• Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAs
• SEQR: Secure and Efficient QR-based LoRA Routing
• Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules
• LoRA-Augmented Generation (LAG) for Knowledge-Intensive Language Tasks
• FLoE: Fisher-Based Layer Selection for Efficient Sparse Adaptation of Low-Rank Experts
• One Rank at a Time: Cascading Error Dynamics in Sequential Learning
• GenKnowSub: Improving Modularity and Reusability of LLMs through General Knowledge Subtraction

Related

• Towards Optimal Adapter Placement for Efficient Transfer Learning
• Parameter-Efficient Multi-Task Learning via Progressive Task-Specific Adaptation
• OrchMoE: Efficient Multi-Adapter Learning with Task-Skill Synergy
• TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts
• Structural Priors and Modular Adapters in the Composable Fine-Tuning Algorithm of Large-Scale Models