Efficiently Distilling LLMs for Edge Applications

2024/04/01 by Achintya Kundu, Kundu, Achintya, Fabian Lim +9 · 1 citation
Engineering · Computer Science · Materials Science · #Advancements in Photolithography Techniques #Advanced Data Storage Technologies #Copper Interconnects and Reliability

paper · pdf · doi:10.48550/arxiv.2404.01353

Abstract

Supernet training of LLMs is of great interest in industrial applications as it confers the ability to produce a palette of smaller models at constant cost, regardless of the number of models (of different size / latency) produced. We propose a new method called Multistage Low-rank Fine-tuning of Super-transformers (MLFS) for parameter-efficient supernet training. We show that it is possible to obtain high-quality encoder models that are suitable for commercial edge applications, and that while decoder-only models are resistant to a comparable degree of compression, decoders can be effectively sliced for a significant reduction in training time.

Cited by

• T-SAR: A Full-Stack Co-design for CPU-Only Ternary LLM Inference via In-Place SIMD ALU Reorganization

Related

• Constrained Edge AI Deployment: Fine-Tuning vs Distillation for LLM Compression
• Efficient Supernet Training with Orthogonal Softmax for Scalable ASR Model Compression
• TwIST: Rigging the Lottery in Transformers with Independent Subnetwork Training
• Investigating Low-Rank Training in Transformer Language Models: Efficiency and Scaling Analysis
• Building on Efficient Foundations: Effectively Training LLMs with Structured Feedforward Layers