Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning

2020/04/08 by Zhaojiang Lin, Andrea Madotto, Lin, Zhaojiang +3 · 19 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.2004.03829

Accepted as Findings of EMNLP 2020, Zhaojiang Lin and Andrea Madotto contributed equally to this work

openalex publication_date 2020/04/08 · arxiv created 2020/09/21 · arxiv updated 2020/09/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Fine-tuning pre-trained generative language models to down-stream language generation tasks has shown promising results. However, this comes with the cost of having a single, large model for each task, which is not ideal in low-memory/power scenarios (e.g., mobile). In this paper, we propose an effective way to fine-tune multiple down-stream generation tasks simultaneously using a single, large pre-trained model. The experiments on five diverse language generation tasks show that by just using an additional 2-3% parameters for each task, our model can maintain or even improve the performance of fine-tuning the whole model.

Citations

• Sequence Level Training with Recurrent Neural Networks
• Pay Less Attention with Lightweight and Dynamic Convolutions
• LLM-Supported Manufacturing Mapping Generation

Cited by

• Towards Efficient Post-Training via Fourier-Driven Adapter Architectures
• Can abstract concepts from LLM improve SLM performance?
• TuckA: Hierarchical Compact Tensor Experts for Efficient Fine-Tuning
• GaLLoP: Gradient-based Sparse Learning on Low-Magnitude Parameters
• NeuroAda: Activating Each Neuron's Potential for Parameter-Efficient Fine-Tuning
• Long Exposure: Accelerating Parameter-Efficient Fine-Tuning for LLMs under Shadowy Sparsity
• Teamwork: Collaborative Diffusion with Low-rank Coordination and Adaptation
• Optimizing Fine-Tuning through Advanced Initialization Strategies for Low-Rank Adaptation
• LoRAFusion: Efficient LoRA Fine-Tuning for LLMs
• Few-Shot Bot: Prompt-Based Learning for Dialogue Systems
• GeoRA: Geometry-Aware Low-Rank Adaptation for RLVR
• MaCP: Minimal yet Mighty Adaptation via Hierarchical Cosine Projection
• SC-LoRA: Balancing Efficient Fine-tuning and Knowledge Preservation via Subspace-Constrained LoRA
• UORA: Uniform Orthogonal Reinitialization Adaptation in Parameter-Efficient Fine-Tuning of Large Models
• RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models
• Language Model Networks: Supervision-Efficient Learning through Dense Communication
• Quantum-Enhanced Parameter-Efficient Learning for Typhoon Trajectory Forecasting
• A Systematic Literature Review of Parameter-Efficient Fine-Tuning for Large Code Models
• Prompt-Tuning SAM: From Generalist to Specialist with only 2048 Parameters and 16 Training Images
• Large language model for post‐earthquake structural damage assessment of buildings

Related

• Making Small Language Models Better Multi-task Learners with Mixture-of-Task-Adapters
• LlamaTurk: Adapting Open-Source Generative Large Language Models for Low-Resource Language
• Data-Efficiency with a Single GPU: An Exploration of Transfer Methods for Small Language Models
• Key ingredients for effective zero-shot cross-lingual knowledge transfer in generative tasks
• GLaM: Efficient Scaling of Language Models with Mixture-of-Experts