2021/04/01 by Zhiqiang Shen, Shen, Zhiqiang, Zechun Liu +9 · 2 citations
Computer Science · #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification
paper · pdf · doi:10.48550/arxiv.2104.00676
openalex publication_date 2021/04/01 · openalex created_date 2021/04/13 · openalex updated_date 2026/07/28
This work aims to empirically clarify a recently discovered perspective that\nlabel smoothing is incompatible with knowledge distillation. We begin by\nintroducing the motivation behind on how this incompatibility is raised, i.e.,\nlabel smoothing erases relative information between teacher logits. We provide\na novel connection on how label smoothing affects distributions of semantically\nsimilar and dissimilar classes. Then we propose a metric to quantitatively\nmeasure the degree of erased information in sample's representation. After\nthat, we study its one-sidedness and imperfection of the incompatibility view\nthrough massive analyses, visualizations and comprehensive experiments on Image\nClassification, Binary Networks, and Neural Machine Translation. Finally, we\nbroadly discuss several circumstances wherein label smoothing will indeed lose\nits effectiveness. Project page:\nhttp://zhiqiangshen.com/projects/LSandKD/index.html.\n