Compacter: Efficient Low-Rank Hypercomplex Adapter Layers

2021/06/08 by Rabeeh Karimi Mahabadi, Mahabadi, Rabeeh Karimi, James Henderson +3 · 36 citations
Computer Science · #Topic Modeling #Natural Language Processing Techniques #Domain Adaptation and Few-Shot Learning

paper · pdf · doi:10.48550/arxiv.2106.04647

Abstract

Adapting large-scale pretrained language models to downstream tasks via fine-tuning is the standard method for achieving state-of-the-art performance on NLP benchmarks. However, fine-tuning all weights of models with millions or billions of parameters is sample-inefficient, unstable in low-resource settings, and wasteful as it requires storing a separate copy of the model for each task. Recent work has developed parameter-efficient fine-tuning methods, but these approaches either still require a relatively large number of parameters or underperform standard fine-tuning. In this work, we propose Compacter, a method for fine-tuning large-scale language models with a better trade-off between task performance and the number of trainable parameters than prior work. Compacter accomplishes this by building on top of ideas from adapters, low-rank optimization, and parameterized hypercomplex multiplication layers. Specifically, Compacter inserts task-specific weight matrices into a pretrained model's weights, which are computed efficiently as a sum of Kronecker products between shared "slow" weights and "fast" rank-one matrices defined per Compacter layer. By only training 0.047% of a pretrained model's parameters, Compacter performs on par with standard fine-tuning on GLUE and outperforms standard fine-tuning on SuperGLUE and low-resource settings. Our code is publicly available at~\urlhttps://github.com/rabeehk/compacter.

Citations

• True Few-Shot Learning with Language Models

Cited by

• Interpretable Safety Alignment via SAE-Constructed Low-Rank Subspace Adaptation
• Towards Efficient Post-Training via Fourier-Driven Adapter Architectures
• Simplifying Multi-Task Architectures Through Task-Specific Normalization
• Governance-Aware Hybrid Fine-Tuning for Multilingual Large Language Models
• Per-Axis Weight Deltas for Frequent Model Updates
• Beyond Real Weights: Hypercomplex Representations for Stable Quantization
• PVeRA: Probabilistic Vector-Based Random Matrix Adaptation
• LUNE: Efficient LLM Unlearning via LoRA Fine-Tuning with Negative Examples
• Serving Heterogeneous LoRA Adapters in Distributed LLM Inference Systems
• MoLT: Mixture of Layer-Wise Tokens for Efficient Audio-Visual Learning
• Advanced Black-Box Tuning of Large Language Models with Limited API Calls
• Adaptation of Foundation Models for Medical Image Analysis: Strategies, Challenges, and Future Directions
• Large Language Models Can Be Strong Differentially Private Learners
• Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training
• A Parameter-Efficient Multi-Scale Convolutional Adapter for Synthetic Speech Detection
• LoRA-DA: Data-Aware Initialization for Low-Rank Adaptation via Asymptotic Analysis
• COLA: Continual Learning via Autoencoder Retrieval of Adapters
• Restoring Pruned Large Language Models via Lost Component Compensation
• VeFA: Vector-Based Feature Space Adaptation for Robust Model Fine-Tuning
• A Guardrail for Safety Preservation: When Safety-Sensitive Subspace Meets Harmful-Resistant Null-Space
• Sparse Subnetwork Enhancement for Underrepresented Languages in Large Language Models
• HoRA: Cross-Head Low-Rank Adaptation with Joint Hypernetworks
• MapGenerator: a framework for learning a diffusion model for text promptable map generation
• Context Parametrization with Compositional Adapters
• Training Neural Networks with Fixed Sparse Masks
• Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models
• Singular Value Few-shot Adaptation of Vision-Language Models
• VaccineRAG: Boosting Multimodal Large Language Models' Immunity to Harmful RAG Samples
• GEM: A Scale-Aware and Distribution-Sensitive Sparse Fine-Tuning Framework for Effective Downstream Adaptation
• MEPT: Mixture of Expert Prompt Tuning as a Manifold Mapper
• HingeNet: A Harmonic-Aware Fine-Tuning Approach for Beat Tracking
• TiMoE: Time-Aware Mixture of Language Experts
• A Bayesian Hybrid Parameter-Efficient Fine-Tuning Method for Large Language Models
• Multi-Task Dense Prediction Fine-Tuning with Mixture of Fine-Grained Experts
• UPP: Unified Point-Level Prompting for Robust Point Cloud Analysis
• Hybrid and Unitary PEFT for Resource-Efficient Large Language Models

Related

• MELoRA: Mini-Ensemble Low-Rank Adapters for Parameter-Efficient Fine-Tuning
• DenseLoRA: Dense Low-Rank Adaptation of Large Language Models
• CoLA: Collaborative Low-Rank Adaptation
• LoRA-Mini : Adaptation Matrices Decomposition and Selective Training
• FlexiGPT: Pruning and Extending Large Language Models with Low-Rank Weight Sharing