Deep Weight Factorization: Sparse Learning Through the Lens of Artificial Symmetries

2025/02/04 by Chris Kolb, Kolb, Chris, Tobias Weber +5 · 1 voice · 2 citations
Computer Science · Engineering · Social Sciences · #Advanced Computing and Algorithms #Gait Recognition and Analysis #Hand Gesture Recognition Systems #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2502.02496

openalex publication_date 2025/02/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Sparse regularization techniques are well-established in machine learning, yet their application in neural networks remains challenging due to the non-differentiability of penalties like the L₁ norm, which is incompatible with stochastic gradient descent. A promising alternative is shallow weight factorization, where weights are decomposed into two factors, allowing for smooth optimization of L₁-penalized neural networks by adding differentiable L₂ regularization to the factors. In this work, we introduce deep weight factorization, extending previous shallow approaches to more than two factors. We theoretically establish equivalence of our deep factorization with non-convex sparse regularization and analyze its impact on training dynamics and optimization. Due to the limitations posed by standard training practices, we propose a tailored initialization scheme and identify important learning rate requirements necessary for training factorized networks. We demonstrate the effectiveness of our deep weight factorization through experiments on various architectures and datasets, consistently outperforming its shallow counterpart and widely used pruning methods.

Cited by

• Differentiable Sparsity via D-Gating: Simple and Versatile Structured Penalization
• WEEP: A Differentiable Nonconvex Sparse Regularizer via Weakly-Convex Envelope

Discussions

• Have you ever wondered how to incorporate (non-)convex L_{2/D}, D>=2, regularization into your neural network while still optimizing it with SGD --- despite the non-smoothness of the penalty? Check ou [bsky, 7 points, 2 comments]

Related

• Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks
• Adaptive Low-Rank Factorization to regularize shallow and deep neural networks
• Decoupled Weight Decay for Any p Norm
• Learning Sparse Neural Networks through L₀ Regularization
• Transformed ℓ₁ Regularization for Learning Sparse Deep Neural Networks