Theoretical Properties for Neural Networks with Weight Matrices of Low Displacement Rank

2017/03/01 by Liang Zhao, Zhao, Liang, Siyu Liao +11 · 2 citations
Computer Science · Engineering · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Face and Expression Recognition #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and ELM #Neural Networks and Applications #Structural Health Monitoring Techniques

paper · pdf · doi:10.48550/arxiv.1703.00144

openalex publication_date 2017/03/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recently low displacement rank (LDR) matrices, or so-called structured matrices, have been proposed to compress large-scale neural networks. Empirical results have shown that neural networks with weight matrices of LDR matrices, referred as LDR neural networks, can achieve significant reduction in space and computational complexity while retaining high accuracy. We formally study LDR matrices in deep learning. First, we prove the universal approximation property of LDR neural networks with a mild condition on the displacement operators. We then show that the error bounds of LDR neural networks are as efficient as general neural networks with both single-layer and multiple-layer structure. Finally, we propose back-propagation based training algorithm for general LDR neural networks.

Cited by

• Separable Layers Enable Structured Efficient Linear Substitutions
• Exploring Weight Symmetry in Deep Neural Networks

Related

• Learning Compressed Transforms with Low Displacement Rank
• Differentiable Learning of Generalized Structured Matrices for Efficient Deep Neural Networks
• An Overview of Low-Rank Structures in the Training and Adaptation of Large Models
• On Generalization Bounds for Neural Networks with Low Rank Layers
• Compressed Deep Networks: Goodbye SVD, Hello Robust Low-Rank Approximation