Using Language Models on Low-end Hardware

2023/05/03 by Ziegner, Fabian, Borst, Janos, Niekler, Andreas +1
#Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2305.02350

Abstract

This paper evaluates the viability of using fixed language models for training text classification networks on low-end hardware. We combine language models with a CNN architecture and put together a comprehensive benchmark with 8 datasets covering single-label and multi-label classification of topic, sentiment, and genre. Our observations are distilled into a list of trade-offs, concluding that there are scenarios, where not fine-tuning a language model yields competitive effectiveness at faster training, requiring only a quarter of the memory compared to fine-tuning.

Related

• Performance-Efficiency Trade-Offs in Adapting Language Models to Text Classification Tasks
• The Importance of Context in Very Low Resource Language Modeling
• Small Language Models in the Real World: Insights from Industrial Text Classification
• Progress and Tradeoffs in Neural Language Models
• Small Models, Big Impact: Efficient Corpus and Graph-Based Adaptation of Small Multilingual Language Models for Low-Resource Languages