Dynamical Low-Rank Compression of Neural Networks with Robustness under Adversarial Attacks

2025/05/12 by Schotthöfer, Steffen, Yang, H. Lexie, Schnake, Stefan · 2 citations
#FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Numerical Analysis (math.NA)

paper · doi:10.48550/arxiv.2505.08022

Abstract

Deployment of neural networks on resource-constrained devices demands models that are both compact and robust to adversarial inputs. However, compression and adversarial robustness often conflict. In this work, we introduce a dynamical low-rank training scheme enhanced with a novel spectral regularizer that controls the condition number of the low-rank core in each layer. This approach mitigates the sensitivity of compressed models to adversarial perturbations without sacrificing accuracy on clean data. The method is model- and data-agnostic, computationally efficient, and supports rank adaptivity to automatically compress the network at hand. Extensive experiments across standard architectures, datasets, and adversarial attacks show the regularized networks can achieve over 94% compression while recovering or improving adversarial accuracy relative to uncompressed baselines.

Cited by

• StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold
• Mitigating Subject Dependency in EEG Decoding with Subject-Specific Low-Rank Adapters

Related

• Compression-aware Training of Deep Networks
• On the Effect of Low-Rank Weights on Adversarial Robustness of Neural Networks
• Theoretical Guarantees for Low-Rank Compression of Deep Neural Networks
• An Empirical Study of the Influence of Adversarial Fine-Tuning on Compressed Neural Networks
• Adversarial Robustness vs Model Compression, or Both?