Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptation

2024/06/06 by Can Yaras, Peng Wang, Yaras, Can +5 · 6 citations
Computer Science · Physics and Astronomy · #Anomaly Detection Techniques and Applications #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #FOS: Electrical engineering #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Model Reduction and Neural Networks #Signal Processing (eess.SP) #electronic engineering #information engineering

paper · pdf · doi:10.48550/arxiv.2406.04112

openalex publication_date 2024/06/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

While overparameterization in machine learning models offers great benefits in terms of optimization and generalization, it also leads to increased computational requirements as model sizes grow. In this work, we show that by leveraging the inherent low-dimensional structures of data and compressible dynamics within the model parameters, we can reap the benefits of overparameterization without the computational burdens. In practice, we demonstrate the effectiveness of this approach for deep low-rank matrix completion as well as fine-tuning language models. Our approach is grounded in theoretical findings for deep overparameterized low-rank matrix recovery, where we show that the learning dynamics of each weight matrix are confined to an invariant low-dimensional subspace. Consequently, we can construct and train compact, highly compressed factorizations possessing the same benefits as their overparameterized counterparts. In the context of deep matrix completion, our technique substantially improves training efficiency while retaining the advantages of overparameterization. For language model fine-tuning, we propose a method called "Deep LoRA", which improves the existing low-rank adaptation (LoRA) technique, leading to reduced overfitting and a simplified hyperparameter setup, while maintaining comparable efficiency. We validate the effectiveness of Deep LoRA on natural language tasks, particularly when fine-tuning with limited data. Our code is available at https://github.com/cjyaras/deep-lora-transformers.

Cited by

• On the Convergence Rate of LoRA Gradient Descent
• Coarse-to-Fine Hierarchical Alignment for UAV-based Human Detection using Diffusion Models
• Randomized Gradient Subspaces for Efficient Large Language Model Training
• Understanding Incremental Learning with Closed-form Solution to Gradient Flow on Overparamerterized Matrix Factorization
• Unpacking the Implicit Norm Dynamics of Sharpness-Aware Minimization in Tensorized Models
• Beyond Value Functions: Single-Loop Bilevel Optimization under Flatness Conditions

Related

• Efficient Compression of Overparameterized Deep Models through Low-Dimensional Learning Dynamics
• OP-LoRA: The Blessing of Dimensionality
• Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matrices
• MLorc: Momentum Low-rank Compression for Memory Efficient Large Language Model Adaptation
• CoRA: Optimizing Low-Rank Adaptation with Common Subspace of Large Language Models