An Optimization and Generalization Analysis for Max-Pooling Networks

2020/02/22 by Brutzkus, Alon, Globerson, Amir
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.2002.09781

Abstract

Max-Pooling operations are a core component of deep learning architectures. In particular, they are part of most convolutional architectures used in machine vision, since pooling is a natural approach to pattern detection problems. However, these architectures are not well understood from a theoretical perspective. For example, we do not understand when they can be globally optimized, and what is the effect of over-parameterization on generalization. Here we perform a theoretical analysis of a convolutional max-pooling architecture, proving that it can be globally optimized, and can generalize well even for highly over-parameterized models. Our analysis focuses on a data generating distribution inspired by pattern detection problem, where a "discriminative" pattern needs to be detected among "spurious" patterns. We empirically validate that CNNs significantly outperform fully connected networks in our setting, as predicted by our theoretical results.

Related

• The Theoretical Expressiveness of Maxpooling
• Comparison of Methods Generalizing Max- and Average-Pooling
• Generalizing Pooling Functions in Convolutional Neural Networks: Mixed, Gated, and Tree
• Max-Pooling Dropout for Regularization of Convolutional Neural Networks
• Sorted Pooling in Convolutional Networks for One-shot Learning