NASH: A Simple Unified Framework of Structured Pruning for Accelerating Encoder-Decoder Language Models

2023/10/16 by Ko, Jongwoo, Park, Seungjoon, Kim, Yujin +4 · 1 citation
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2310.10054

Abstract

Structured pruning methods have proven effective in reducing the model size and accelerating inference speed in various network architectures such as Transformers. Despite the versatility of encoder-decoder models in numerous NLP tasks, the structured pruning methods on such models are relatively less explored compared to encoder-only models. In this study, we investigate the behavior of the structured pruning of the encoder-decoder models in the decoupled pruning perspective of the encoder and decoder component, respectively. Our findings highlight two insights: (1) the number of decoder layers is the dominant factor of inference speed, and (2) low sparsity in the pruned encoder network enhances generation quality. Motivated by these findings, we propose a simple and effective framework, NASH, that narrows the encoder and shortens the decoder networks of encoder-decoder models. Extensive experiments on diverse generation and inference tasks validate the effectiveness of our method in both speedup and output quality.

Cited by

• MaskPro: Linear-Space Probabilistic Learning for Strict (N:M)-Sparsity on LLMs

Related

• On Sparsifying Encoder Outputs in Sequence-to-Sequence Models
• Rethinking Network Pruning -- under the Pre-train and Fine-tune Paradigm
• 2SSP: A Two-Stage Framework for Structured Pruning of LLMs
• FinerCut: Finer-grained Interpretable Layer Pruning for Large Language Models
• Less is KEN: a Universal and Simple Non-Parametric Pruning Algorithm for Large Language Models