Towards Robust Low-Resource Fine-Tuning with Multi-View Compressed Representations

2022/11/16 by Liu, Linlin, Li, Xingxuan, Thakkar, Megh +4 · 3 citations
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2211.08794

Abstract

Due to the huge amount of parameters, fine-tuning of pretrained language models (PLMs) is prone to overfitting in the low resource scenarios. In this work, we present a novel method that operates on the hidden representations of a PLM to reduce overfitting. During fine-tuning, our method inserts random autoencoders between the hidden layers of a PLM, which transform activations from the previous layers into multi-view compressed representations before feeding them into the upper layers. The autoencoders are plugged out after fine-tuning, so our method does not add extra parameters or increase computation cost during inference. Our method demonstrates promising performance improvement across a wide range of sequence- and token-level low-resource NLP tasks.

Cited by

• Selecting Auxiliary Data via Neural Tangent Kernels for Low-Resource Domains
• Exploring Kolmogorov-Arnold Network Expansions in Vision Transformers for Mitigating Catastrophic Forgetting in Continual Learning
• IM-BERT: Enhancing Robustness of BERT through the Implicit Euler Method

Related

• Parameter-Efficient Fine-Tuning without Introducing New Latency
• Just CHOP: Embarrassingly Simple LLM Compression
• Small Language Models Improve Giants by Rewriting Their Outputs
• Enhancing Inference Efficiency of Large Language Models: Investigating Optimization Strategies and Architectural Innovations
• General Purpose Text Embeddings from Pre-trained Language Models for Scalable Inference