Separable Layers Enable Structured Efficient Linear Substitutions

2019/06/03 by Gavin Gray, Gray, Gavin, Elliot J. Crowley +3
Computer Science · Physics and Astronomy · #Cellular Automata and Applications #Coding theory and cryptography #Electromagnetic Scattering and Analysis #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · pdf · doi:10.48550/arxiv.1906.00859

openalex publication_date 2019/06/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In response to the development of recent efficient dense layers, this paper shows that something as simple as replacing linear components in pointwise convolutions with structured linear decompositions also produces substantial gains in the efficiency/accuracy tradeoff. Pointwise convolutions are fully connected layers and are thus prepared for replacement by structured transforms. Networks using such layers are able to learn the same tasks as those using standard convolutions, and provide Pareto-optimal benefits in efficiency/accuracy, both in terms of computation (mult-adds) and parameter count (and hence memory). Code is available at https://github.com/BayesWatch/deficient-efficient.

Citations

• Tensorizing Neural Networks
• ImageNet Large Scale Visual Recognition Challenge
• Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer
• MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications
• DARTS: Differentiable Architecture Search
• ChannelNets: Compact and Efficient Convolutional Neural Networks via Channel-Wise Convolutions
• Theoretical Properties for Neural Networks with Weight Matrices of Low Displacement Rank
• Learning Multiple Layers of Features from Tiny Images

Related

• Structured Convolutions for Efficient Neural Network Design
• More is Less: A More Complicated Network with Less Inference Complexity
• An Equivalence of Fully Connected Layer and Convolutional Layer
• The Singular Values of Convolutional Layers
• How far can we go without convolution: Improving fully-connected networks