COALA: Numerically Stable and Efficient Framework for Context-Aware Low-Rank Approximation

2025/07/10 by Uliana Parkina, Parkina, Uliana, Maxim Rakhuba +1 · 2 citations
Computer Science · Engineering · Mathematics · #65F55 #68T50 #Computation and Language (cs.CL) #FOS: Computer and information sciences #FOS: Mathematics #Machine Learning (cs.LG) #Numerical Analysis (math.NA) #Sparse and Compressive Sensing Techniques #Stochastic Gradient Optimization Techniques #Tensor decomposition and applications

paper · pdf · doi:10.48550/arxiv.2507.07580

openalex publication_date 2025/07/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent studies suggest that context-aware low-rank approximation is a useful tool for compression and fine-tuning of modern large-scale neural networks. In this type of approximation, a norm is weighted by a matrix of input activations, significantly improving metrics over the unweighted case. Nevertheless, existing methods for neural networks suffer from numerical instabilities due to their reliance on classical formulas involving explicit Gram matrix computation and their subsequent inversion. We demonstrate that this can degrade the approximation quality or cause numerically singular matrices. To address these limitations, we propose a novel inversion-free regularized framework that is based entirely on stable decompositions and overcomes the numerical pitfalls of prior art. Our method can handle possible challenging scenarios: (1) when calibration matrices exceed GPU memory capacity, (2) when input activation matrices are nearly singular, and even (3) when insufficient data prevents unique approximation. For the latter, we prove that our solution converges to a desired approximation and derive explicit error bounds.

Citations

• Matrix-Free Two-to-Infinity and One-to-Two Norms Estimation
• SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compression
• AdaSVD: Adaptive Singular Value Decomposition for Large Language Models
• The Llama 3 Herd of Models
• LoRA-GA: Low-Rank Adaptation with Gradient Approximation
• BlockPruner: Fine-grained Pruning for Large Language Models
• Group and Shuffle: Efficient Structured Orthogonal Parametrization
• MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning
• CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning
• A Survey on Efficient Inference for Large Language Models
• PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models
• SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression
• BiLLM: Pushing the Limit of Post-Training Quantization for LLMs
• SliceGPT: Compress Large Language Models by Deleting Rows and Columns
• Fluctuation-based Adaptive Structured Pruning for Large Language Models
• ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models
• Efficient Large Language Models: A Survey
• PB-LLM: Partially Binarized Large Language Models
• A Survey on Model Compression for Large Language Models
• A Simple and Effective Pruning Approach for Large Language Models
• LLM-Pruner: On the Structural Pruning of Large Language Models
• A Survey of Large Language Models
• SVDiff: Compact Parameter Space for Diffusion Fine-Tuning
• DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
• DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation
• Language model compression with weighted low-rank factorization
• LoRA: Low-Rank Adaptation of Large Language Models
• Measuring Massive Multitask Language Understanding
• On the Global Convergence of Training Deep Linear ResNets
• PIQA: Reasoning about Physical Commonsense in Natural Language
• BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
• HellaSwag: Can a Machine Really Finish Your Sentence?
• Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• A Survey of Model Compression and Acceleration for Deep Neural Networks
• Data-Free Knowledge Distillation for Deep Neural Networks
• Pointer Sentinel Mixture Models
• Random perturbation of low rank matrices: Improving classical bounds
• Communication-optimal parallel and sequential QR and LU factorizations

Cited by

• Globally optimized SVD compression of LLMs via Fermi-function-based rank selection and gauge fixing
• LoRA meets Riemannion: Muon Optimizer for Parametrization-independent Low-Rank Adapters

Related

• Compressed Deep Networks: Goodbye SVD, Hello Robust Low-Rank Approximation
• Approximation Power of Deep Neural Networks: an explanatory mathematical survey
• Theoretical Guarantees for Low-Rank Compression of Deep Neural Networks
• Low-Rank Approximations for Conditional Feedforward Computation in Deep Neural Networks
• Approximation bounds for norm constrained deep neural networks