Scattering Networks for Hybrid Representation Learning

2018/07/19 by Edouard Oyallon, Sergey Zagoruyko, Gabriel Huang +5 · 81 citations
Computer Science · Mathematics · #AI in cancer detection #Advanced Neural Network Applications #Algorithm #Artificial intelligence #Artificial neural network #Competitive learning #Computer science #Convolutional neural network #Deep learning #Domain Adaptation and Few-Shot Learning #Feature learning #Machine learning #Optics #Pattern recognition (psychology) #Physics #Representation (politics) #Residual #Scattering #Stability (learning theory) #Supervised learning #Unsupervised learning #cs.CV #cs.LG #stat.ML

paper · pdf · doi:10.1109/tpami.2018.2855738

published in IEEE Transactions on Pattern Analysis and Machine Intelligence 41(9), 2208-2221 (IEEE Computer Society) · arXiv admin note: substantial text overlap with arXiv:1703.08961

openalex publication_date 2018/07/19 · arxiv created 2018/09/17 · arxiv updated 2018/09/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Scattering networks are a class of designed Convolutional Neural Networks (CNNs) with fixed weights. We argue they can serve as generic representations for modelling images. In particular, by working in scattering space, we achieve competitive results both for supervised and unsupervised learning tasks, while making progress towards constructing more interpretable CNNs. For supervised learning, we demonstrate that the early layers of CNNs do not necessarily need to be learned, and can be replaced with a scattering network instead. Indeed, using hybrid architectures, we achieve the best results with predefined representations to-date, while being competitive with end-to-end learned CNNs. Specifically, even applying a shallow cascade of small-windowed scattering coefficients followed by 1× 11×1-convolutions results in AlexNet accuracy on the ILSVRC2012 classification task. Moreover, by combining scattering networks with deep residual networks, we achieve a single-crop top-5 error of 11.4 percent on ILSVRC2012. Also, we show they can yield excellent performance in the small sample regime on CIFAR-10 and STL-10 datasets, exceeding their end-to-end counterparts, through their ability to incorporate geometrical priors. For unsupervised learning, scattering coefficients can be a competitive representation that permits image recovery. We use this fact to train hybrid GANs to generate images. Finally, we empirically analyze several properties related to stability and reconstruction of images from scattering coefficients.

Citations

• Scaling the Scattering Transform: Deep Hybrid Networks
• Unsupervised Visual Representation Learning by Context Prediction
• Stacked What-Where Auto-encoders
• Audio Texture Synthesis with Scattering Moments
• Intriguing properties of neural networks
• Learning Stable Group Invariant Representations with Convolutional Networks
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Explaining and Harnessing Adversarial Examples
• Approximation by superpositions of a sigmoidal function
• Striving for Simplicity: The All Convolutional Net
• Convolutional Kernel Networks
• A theory for multiresolution signal decomposition: the wavelet representation
• ImageNet classification with deep convolutional neural networks
• Understanding Deep Convolutional Networks
• Wide Residual Networks
• Adversarial Feature Learning
• What makes ImageNet good for transfer learning?
• Deep unsupervised learning through spatial contrasting
• Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks
• Highway Networks

Cited by

• Bottom-Up Scattering Information Perception Network for SAR target recognition