Theoretical Guarantees for Low-Rank Compression of Deep Neural Networks

2025/02/04 by Shihao Zhang, Rayan Saab, Zhang, Shihao +1 · 2 citations
Computer Science · #Anomaly Detection Techniques and Applications #FOS: Computer and information sciences #Information Theory (cs.IT) #Machine Learning (cs.LG) #Neural Networks and Applications

paper · pdf · doi:10.48550/arxiv.2502.02766

openalex publication_date 2025/02/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/31

Abstract

Deep neural networks have achieved state-of-the-art performance across numerous applications, but their high memory and computational demands present significant challenges, particularly in resource-constrained environments. Model compression techniques, such as low-rank approximation, offer a promising solution by reducing the size and complexity of these networks while only minimally sacrificing accuracy. In this paper, we develop an analytical framework for data-driven post-training low-rank compression. We prove three recovery theorems under progressively weaker assumptions about the approximate low-rank structure of activations, modeling deviations via noise. Our results represent a step toward explaining why data-driven low-rank compression methods outperform data-agnostic approaches and towards theoretically grounded compression algorithms that reduce inference costs while maintaining performance.

Cited by

• Beacon: Post-Training Quantization with Integrated Grid Selection
• Provable Post-Training Quantization: Theoretical Analysis of OPTQ and Qronos

Related

• Data-Driven Low-Rank Neural Network Compression
• Compression-aware Training of Deep Networks
• Dynamical Low-Rank Compression of Neural Networks with Robustness under Adversarial Attacks
• Learning Low-Rank Approximation for CNNs
• Rate Distortion For Model Compression: From Theory To Practice