Everything old is new again: A multi-view learning approach to learning using privileged information and distillation

2019/03/08 by Weiran Wang, Wang, Weiran
Computer Science · Decision Sciences · Mathematics · #Advanced Bandit Algorithms Research #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Algorithms #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1903.03694

arxiv created 2019/03/08 · openalex publication_date 2019/03/08 · arxiv updated 2019/03/12 · openalex created_date 2019/03/22 · openalex updated_date 2026/07/28

Abstract

We adopt a multi-view approach for analyzing two knowledge transfer settings---learning using privileged information (LUPI) and distillation---in a common framework. Under reasonable assumptions about the complexities of hypothesis spaces, and being optimistic about the expected loss achievable by the student (in distillation) and a transformed teacher predictor (in LUPI), we show that encouraging agreement between the teacher and the student leads to reduced search space. As a result, improved convergence rate can be obtained with regularized empirical risk minimization.

Citations

• Optimistic Rates for Learning with a Smooth Loss
• Distilling the Knowledge in a Neural Network
• Do Deep Nets Really Need to be Deep?
• Doubly Accelerated Methods for Faster CCA and Generalized Eigendecomposition

Related

• Cooperative Knowledge Distillation: A Learner Agnostic Approach
• [Re] Distilling Knowledge via Knowledge Review
• Unifying distillation and privileged information
• Why distillation helps: a statistical perspective
• Improving Knowledge Distillation via Transferring Learning Ability