2022/11/18 by Khalifa, Amr, Michael C. Mozer, Mozer, Michael C. +6
Computer Science · Materials Science · Physics and Astronomy · #FOS: Computer and information sciences #I.2.10 #I.2.6 #Machine Learning (cs.LG) #Machine Learning in Materials Science #Model Reduction and Neural Networks #Neural Networks and Applications
paper · pdf · doi:10.48550/arxiv.2211.10193
openalex publication_date 2022/11/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Recent works demonstrate that early layers in a neural network contain useful information for prediction. Inspired by this, we show that extending temperature scaling across all layers improves both calibration and accuracy. We call this procedure "layer-stack temperature scaling" (LATES). Informally, LATES grants each layer a weighted vote during inference. We evaluate it on five popular convolutional neural network architectures both in- and out-of-distribution and observe a consistent improvement over temperature scaling in terms of accuracy, calibration, and AUC. All conclusions are supported by comprehensive statistical analyses. Since LATES neither retrains the architecture nor introduces many more parameters, its advantages can be reaped without requiring additional data beyond what is used in temperature scaling. Finally, we show that combining LATES with Monte Carlo Dropout matches state-of-the-art results on CIFAR10/100.