Transferring Inductive Biases through Knowledge Distillation

2020/05/31 by Samira Abnar, Mostafa Dehghani, Abnar, Samira +3 · 4 citations
Computer Science · Mathematics · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Topic Modeling #cs.AI #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2006.00555

openalex publication_date 2020/05/31 · arxiv created 2020/10/04 · arxiv updated 2020/10/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Having the right inductive biases can be crucial in many tasks or scenarios where data or computing resources are a limiting factor, or where training data is not perfectly representative of the conditions at test time. However, defining, designing and efficiently adapting inductive biases is not necessarily straightforward. In this paper, we explore the power of knowledge distillation for transferring the effect of inductive biases from one model to another. We consider families of models with different inductive biases, LSTMs vs. Transformers and CNNs vs. MLPs, in the context of tasks and scenarios where having the right inductive biases is critical. We study the effect of inductive biases on the solutions the models converge to and investigate how and to what extent the effect of inductive biases is transferred through knowledge distillation, in terms of not only performance but also different aspects of converged solutions.

Citations

• Distilling the Knowledge in a Neural Network
• Long Short-Term Memory
• Finding Structure in Time
• Distilling a Neural Network Into a Soft Decision Tree
• Universal Transformers
• Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
• Theoretical Limitations of Self-Attention in Neural Sequence Models
• Universality and individuality in neural dynamics across large populations of recurrent networks
• Large scale distributed neural network training through online distillation
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• LLM-Supported Manufacturing Mapping Generation
• Language Models are Few-Shot Learners

Cited by

• Propositional Logic for Probing Generalization in Neural Networks
• Self-Distillation Amplifies Regularization in Hilbert Space
• Can RNNs trained on harder subject-verb agreement instances still perform well on easier ones?
• Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning
• Procedural Pretraining: Warming Up Language Models with Abstract Data

Related

• Towards Exact Computation of Inductive Bias
• Teasing Apart Architecture and Initial Weights as Sources of Inductive Bias in Neural Networks
• A Model of Inductive Bias Learning
• Instilling Inductive Biases with Subnetworks
• Interpolated-MLPs: Controllable Inductive Bias