2021/05/17 by Yuxin Wu, Justin Johnson, Wu, Yuxin +1 · 1 voice · 5 citations
Computer Science · #Advanced Neural Network Applications #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning and Algorithms #cs.CV
paper · pdf · doi:10.48550/arxiv.2105.07576
Tech report
arxiv created 2021/05/17 · openalex publication_date 2021/05/17 · arxiv published 2021/05/17 · arxiv updated 2021/05/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
BatchNorm is a critical building block in modern convolutional neural networks. Its unique property of operating on "batches" instead of individual samples introduces significantly different behaviors from most other operations in deep learning. As a result, it leads to many hidden caveats that can negatively impact model's performance in subtle ways. This paper thoroughly reviews such problems in visual recognition tasks, and shows that a key to address them is to rethink different choices in the concept of "batch" in BatchNorm. By presenting these caveats and their mitigations, we hope this review can help researchers use BatchNorm more effectively.