Latent Space Factorization in LoRA

2025/10/22 by Shashi Kumar, Kumar, Shashi, Yacouba Kaloga +7
Computer Science · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Face recognition and analysis #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2510.19640

openalex publication_date 2025/10/22 · openalex created_date 2025/10/24 · openalex updated_date 2026/07/28

Abstract

Low-rank adaptation (LoRA) is a widely used method for parameter-efficient finetuning. However, existing LoRA variants lack mechanisms to explicitly disambiguate task-relevant information within the learned low-rank subspace, potentially limiting downstream performance. We propose Factorized Variational Autoencoder LoRA (FVAE-LoRA), which leverages a VAE to learn two distinct latent spaces. Our novel Evidence Lower Bound formulation explicitly promotes factorization between the latent spaces, dedicating one latent space to task-salient features and the other to residual information. Extensive experiments on text, audio, and image tasks demonstrate that FVAE-LoRA consistently outperforms standard LoRA. Moreover, spurious correlation evaluations confirm that FVAE-LoRA better isolates task-relevant signals, leading to improved robustness under distribution shifts. Our code is publicly available at: https://github.com/idiap/FVAE-LoRA

Citations

• The Llama 3 Herd of Models
• OLoRA: Orthonormal Low-Rank Adaptation of Large Language Models
• PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models
• LoRA+: Efficient Low Rank Adaptation of Large Models
• DoRA: Weight-Decomposed Low-Rank Adaptation
• A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA
• QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models
• Spurious Correlations and Where to Find Them
• Challenges and Opportunities in Improving Worst-Group Generalization in Presence of Spurious Features
• QLoRA: Efficient Finetuning of Quantized LLMs
• Micrograph segmentations for DDEVD
• Segment Anything
• LLM-Adapters: An Adapter Family for Parameter-Efficient Fine-Tuning of Large Language Models
• AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
• Robust Speech Recognition via Large-Scale Weak Supervision
• High-Resolution Image Synthesis with Latent Diffusion Models
• High-Resolution Image Synthesis with Latent Diffusion Models
• Just Train Twice: Improving Group Robustness without Training Group Information
• BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
• The Power of Scale for Parameter-Efficient Prompt Tuning
• GPT Understands, Too
• Learning Transferable Visual Models From Natural Language Supervision
• Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning
• WILDS: A Benchmark of in-the-Wild Distribution Shifts
• Environment Inference for Invariant Learning
• Robustness to Spurious Correlations via Human Annotations
• wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations
• Visual Transformers: Token-based Image Representation and Processing for Computer Vision
• Language Models are Few-Shot Learners
• An Investigation of Why Overparameterization Exacerbates Spurious Correlations
• PIQA: Reasoning about Physical Commonsense in Natural Language
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• HellaSwag: Can a Machine Really Finish Your Sentence?
• Parameter-Efficient Transfer Learning for NLP
• Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Understanding disentangling in β-VAE
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Disentangling by Factorising
• Variational Inference of Disentangled Latent Concepts from Unlabeled Observations
• Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data
• EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification
• Describing Textures in the Wild
• Mitigating Bias in Dataset Distillation
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Prefix-Tuning: Optimizing Continuous Prompts for Generation

Related

• Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape
• MTL-LoRA: Low-Rank Adaptation for Multi-Task Learning
• AFA-LoRA: Enabling Non-Linear Adaptations in LoRA with Activation Function Annealing
• LoRA Learns Less and Forgets Less
• In-Context Meta LoRA Generation