Beyond Low-rank Decomposition: A Shortcut Approach for Efficient On-Device Learning

2025/05/08 by Thi To Loan Nguyen, Aël Quélennec, Nguyen, Le-Trung +5 · 4 citations
Computer Science · #Advanced Neural Network Applications #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #IoT and Edge/Fog Computing #Machine Learning (cs.LG) #Stochastic Gradient Optimization Techniques

paper · pdf · doi:10.48550/arxiv.2505.05086

openalex publication_date 2025/05/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

On-device learning has emerged as a promising direction for AI development, particularly because of its potential to reduce latency issues and mitigate privacy risks associated with device-server communication, while improving energy efficiency. Despite these advantages, significant memory and computational constraints still represent major challenges for its deployment. Drawing on previous studies on low-rank decomposition methods that address activation memory bottlenecks in backpropagation, we propose a novel shortcut approach as an alternative. Our analysis and experiments demonstrate that our method can reduce activation memory usage, even up to 120.09× compared to vanilla training, while also reducing overall training FLOPs up to 1.86× when evaluated on traditional benchmarks.

Citations

• Activation Map Compression through Tensor Decomposition for Deep Learning
• TinyLlama: An Open-Source Small Language Model
• ASVD: Activation-aware Singular Value Decomposition for Compressing Large Language Models
• Towards On-device Learning on the Edge: Ways to Select Neurons to Update under a Budget Constraint
• Sparse Low-rank Adaptation of Pre-trained Language Models
• Efficient On-device Training via Gradient Filtering
• The Forward-Forward Algorithm: Some Preliminary Investigations
• DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation
• On-Device Training Under 256KB Memory
• 0/1 Deep Neural Networks via Block Coordinate Descent
• Carbon Emissions and Large Neural Network Training
• The Computational Limits of Deep Learning
• Deep Learning for Financial Applications : A Survey
• On-Device Machine Learning: An Algorithms and Learning Theory Perspective
• BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
• Federated Learning for Mobile Keyboard Prediction
• Unified Perceptual Parsing for Scene Understanding
• Lipschitz regularity of deep neural networks: analysis and efficient estimation
• The Reversible Residual Network: Backpropagation Without Storing Activations
• Rethinking Atrous Convolution for Semantic Image Segmentation
• Pyramid Scene Parsing Network
• Deep Residual Learning for Image Recognition
• Fully Convolutional Networks for Semantic Segmentation
• AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning

Cited by

• Memory Constrained Dynamic Subnetwork Update for Transfer Learning
• Study of Training Dynamics for Memory-Constrained Fine-Tuning
• Efficient Resource-Constrained Training of Transformers via Subspace Optimization
• LANCE: Low Rank Activation Compression for Efficient On-Device Continual Learning

Related

• Breaking On-device Training Memory Wall: A Systematic Survey
• TinyTL: Reduce Activations, Not Trainable Parameters for Efficient On-Device Learning
• LANCE: Low Rank Activation Compression for Efficient On-Device Continual Learning
• On-Device Machine Learning: An Algorithms and Learning Theory Perspective
• Activation Map Compression through Tensor Decomposition for Deep Learning