Dynamic Corrective Self-Distillation for Better Fine-Tuning of Pretrained Models

2023/12/12 by Ibtihel Amara, Amara, Ibtihel, Vinija Jain +3
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2312.07028

openalex publication_date 2023/12/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We tackle the challenging issue of aggressive fine-tuning encountered during the process of transfer learning of pre-trained language models (PLMs) with limited labeled downstream data. This problem primarily results in a decline in performance on the subsequent task. Inspired by the adaptive boosting method in traditional machine learning, we present an effective dynamic corrective self-distillation (DCS) approach to improve the fine-tuning of the PLMs. Our technique involves performing a self-distillation mechanism where, at each iteration, the student model actively adapts and corrects itself by dynamically adjusting the weights assigned to individual data points. This iterative self-correcting process significantly enhances the overall fine-tuning capability of PLMs, leading to improved performance and robustness. We conducted comprehensive evaluations using the GLUE benchmark demonstrating the efficacy of our method in enhancing the fine-tuning process for various PLMs across diverse downstream tasks.

Related

• Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning
• Dynamic Self-Distillation via Previous Mini-batches for Fine-tuning Small Language Models
• Embarrassingly Simple Self-Distillation Improves Code Generation
• UNDIAL: Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Models
• Self-Taught Self-Correction for Small Language Models