BALF: Budgeted Activation-Aware Low-Rank Factorization for Fine-Tuning-Free Model Compression

2025/09/29 by David González-Martínez, González-Martínez, David
Computer Science · #Advanced Data Compression Techniques #Digital Filter Design and Implementation #FOS: Computer and information sciences #Machine Learning (cs.LG) #Speech and Audio Processing

paper · pdf · doi:10.48550/arxiv.2509.25136

openalex publication_date 2025/09/29 · openalex created_date 2025/10/19 · openalex updated_date 2026/07/28

Abstract

Activation-aware low-rank factorization techniques yield strong compression results but are generally confined to linear layers, while existing whitening-based theory typically makes an implicit full-rank assumption on activations. We introduce a layer representation framework that extends activation-aware factorization beyond linear layers, including standard and grouped convolutions. Within this framework, our whitening-based formulation is more general than prior ones, naturally covering rank-deficient activations, and yields an optimal low-rank projection that attains the reconstruction error of the best low-rank approximation to layer activations. The resulting singular spectrum provides a closed-form per-layer distortion proxy, which we use to allocate per-layer ranks under explicit FLOP or parameter-count budgets via a Lagrangian relaxation with negligible overhead. Together, these components form BALF, an end-to-end pipeline for efficient vision model compression. Across CNNs and vision transformers on CIFAR-10 and ImageNet-1K, BALF generally achieves higher accuracy than SVD-based factorization baselines at matched FLOP or parameter count targets and remains competitive with other fine-tuning-free compression techniques.

Citations

• Q3R: Quadratic Reweighted Rank Regularizer for Effective Low-Rank Training
• Distribution-Aware Tensor Decomposition for Compression of Convolutional Neural Networks
• On Information Geometry and Iterative Optimization in Model Compression: Operator Factorization
• SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compression
• Forget the Data and Fine-Tuning! Just Fold the Network to Compress
• Training-Free Restoration of Pruned Neural Networks
• Dobi-SVD: Differentiable SVD for LLM Compression and Some New Perspectives
• Dense Vision Transformer Compression with Few Samples
• SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression
• ELRT: Efficient Low-Rank Training for Compact Convolutional Neural Networks
• GTP-ViT: Efficient Vision Transformers via Graph-based Token Propagation
• Going Beyond Neural Network Feature Similarity: The Network Feature Complexity and Its Interpretation Using Category Theory
• Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning
• SVD-NAS: Coupling Low-Rank Approximation and Neural Architecture Search
• Practical Network Acceleration with Tiny Sets
• Compressing Neural Networks: Towards Determining the Optimal Layer-wise Decomposition
• Towards Efficient Tensor Decomposition-Based DNN Model Compression with Optimization Framework
• A Survey of Quantization Methods for Efficient Neural Network Inference
• Sparsity in Deep Learning: Pruning and growth for efficient inference\n and training in neural networks
• Training data-efficient image transformers & distillation through attention
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Stable Low-rank Tensor Decomposition for Compression of Convolutional\n Neural Network
• Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Benchmarking Neural Network Robustness to Common Corruptions and Perturbations
• Do ImageNet Classifiers Generalize to ImageNet?
• Quantization and Training of Neural Networks for Efficient\n Integer-Arithmetic-Only Inference
• Attention Is All You Need
• Parseval Networks: Improving Robustness to Adversarial Examples
• Aggregated Residual Transformations for Deep Neural Networks
• Gaussian Error Linear Units (GELUs)
• Deep Residual Learning for Image Recognition
• Accelerating Very Deep Convolutional Networks for Classification and Detection
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Speeding-up Convolutional Neural Networks Using Fine-tuned\n CP-Decomposition
• Speeding up Convolutional Neural Networks with Low Rank Expansions
• Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation
• ImageNet classification with deep convolutional neural networks

Related

• Compression of Recurrent Neural Networks using Matrix Factorization
• Unified Framework for Pre-trained Neural Network Compression via Decomposition and Optimized Rank Selection
• CompressNAS : A Fast and Efficient Technique for Model Compression using Decomposition
• An Adaptive Tensor-Train Decomposition Approach for Efficient Deep Neural Network Compression
• Compression for Better: A General and Stable Lossless Compression Framework