2015/12/19 by Thomas Wiatowski, Helmut Bölcskei, Wiatowski, Thomas +1 · 1 voice · 22 citations
Computer Science · Earth and Planetary Sciences · Mathematics · Physics and Astronomy · #Algorithm #Artificial intelligence #Computer science #Convolutional neural network #Feature extraction #Image and Signal Denoising Methods #Lipschitz continuity #Mathematical analysis #Mathematics #Model Reduction and Neural Networks #Pattern recognition (psychology) #Seismic Imaging and Inversion Techniques #Softmax function #Wavelet #cs.AI #cs.IT #cs.LG #math.FA #math.IT #stat.ML
paper · pdf · doi:10.48550/arxiv.1512.06293
published in arXiv (Cornell University) (Cornell University) · IEEE Transactions on Information Theory, to appear
openalex publication_date 2015/12/19 · arxiv created 2017/10/24 · arxiv updated 2017/10/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Deep convolutional neural networks have led to breakthrough results in numerous practical machine learning tasks such as classification of images in the ImageNet data set, control-policy-learning to play Atari games or the board game Go, and image captioning. Many of these applications first perform feature extraction and then feed the results thereof into a trainable classifier. The mathematical analysis of deep convolutional neural networks for feature extraction was initiated by Mallat, 2012. Specifically, Mallat considered so-called scattering networks based on a wavelet transform followed by the modulus non-linearity in each network layer, and proved translation invariance (asymptotically in the wavelet scale parameter) and deformation stability of the corresponding feature extractor. This paper complements Mallat's results by developing a theory that encompasses general convolutional transforms, or in more technical parlance, general semi-discrete frames (including Weyl-Heisenberg filters, curvelets, shearlets, ridgelets, wavelets, and learned filters), general Lipschitz-continuous non-linearities (e.g., rectified linear units, shifted logistic sigmoids, hyperbolic tangents, and modulus functions), and general Lipschitz-continuous pooling operators emulating, e.g., sub-sampling and averaging. In addition, all of these elements can be different in different network layers. For the resulting feature extractor we prove a translation invariance result of vertical nature in the sense of the features becoming progressively more translation-invariant with increasing network depth, and we establish deformation sensitivity bounds that apply to signal classes such as, e.g., band-limited functions, cartoon functions, and Lipschitz functions.