2025/02/03 by Martin Genzel, Genzel, Martin, Patrick Putzky +13 · 1 voice · 1 citation
Computer Science · Decision Sciences · #Artificial intelligence #Artificial neural network #Compression (physics) #Computer Graphics and Visualization Techniques #Computer science #Data Visualization and Analytics #Descent (aeronautics) #FOS: Computer and information sciences #Geography #Gradient descent #Machine Learning (cs.LG) #Meteorology #Physics #Simulation Techniques and Applications #cs.LG
paper · pdf · doi:10.48550/arxiv.2502.01717
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2025/02/03 · arxiv published 2025/02/03 · openalex created_date 2025/10/10 · arxiv updated 2025/11/08 · openalex updated_date 2026/08/08
The adoption of Foundation Models in resource-constrained environments remains challenging due to their large size and inference costs. A promising way to overcome these limitations is post-training compression, which aims to balance reduced model size against performance degradation. This work presents Any Compression via Iterative Pruning (ACIP), a novel algorithmic approach to determine a compression-performance trade-off from a single stochastic gradient descent run. To achieve parameter efficiency, we use an SVD-reparametrization of linear layers and iteratively prune their singular values with a sparsity-inducing penalty. Importantly, the pruning order of the parameters is used to derive a global score map that allows compressing a model to any target size without re-computation. We evaluate ACIP on a large selection of open-weight LLMs and downstream tasks, demonstrating state-of-the-art results compared to existing factorization-based compression methods. We also show that ACIP seamlessly complements common quantization-based compression techniques.