FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts

2025/10/09 by Heming Zou, Yunliang Zang, Zou, Heming +7 · 2 citations
Computer Science · Engineering · #Advanced Neural Network Applications #Ferroelectric and Negative Capacitance Devices #Advanced Graph Neural Networks

paper · pdf · doi:10.48550/arxiv.2510.08396

Abstract

Low-Rank Adaptation (LoRA) is a widely used parameter-efficient fine-tuning method for foundation models, but it suffers from parameter interference, resulting in suboptimal performance. Although Mixture-of-Experts (MoE)-based LoRA variants show promise in mitigating intra-task correlations in single-task instruction tuning, they introduce additional router parameters and remain ineffective in multi-task model merging where inter-task interference arises. Inspired by the fly olfactory circuit, we propose FlyLoRA, an implicit MoE-based LoRA variant that introduces: (1) rank-wise expert activation in the up-projection matrix, and (2) an implicit router that unifies expert routing and down-projection, where a frozen sparse random projection matrix replaces the traditional dense trainable version. This design resolves the trade-off between intra-task decorrelation and computational efficiency by eliminating the need for an explicit router, while inherently mitigating inter-task interference due to the orthogonality property of random matrices. Extensive experiments across four domains -- general knowledge understanding, scientific question answering, mathematical reasoning, and code generation -- demonstrate consistent performance improvements over existing methods. Beyond empirical gains, FlyLoRA highlights how biological structures can inspire innovations in AI technologies. Code is available at https://github.com/gfyddha/FlyLoRA.

Citations

• Seemingly Redundant Modules Enhance Robust Odor Learning in Fruit Flies
• Utility-Diversity Aware Online Batch Selection for LLM Supervised Fine-tuning
• Fly-CL: A Fly-Inspired Framework for Enhancing Efficient Decorrelation and Reduced Training Time in Pre-trained Model-based Continual Representation Learning
• Group Sequence Policy Optimization
• Can Prompt Difficulty be Online Predicted for Accelerating RL Finetuning of Reasoning Models?
• Fast and Robust: Task Sampling with Posterior and Diversity Synergies for Adaptive Decision-Makers in Randomized Environments
• When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformers
• LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation
• Structural features of the fly olfactory circuit mitigate the stability-plasticity dilemma in continual learning
• DynaPrompt: Dynamic Test-Time Prompt Tuning
• Autonomy-of-Experts Models
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Model Predictive Task Sampling for Efficient and Robust Adaptation
• Qwen2.5 Technical Report
• Doubly Mild Generalization for Offline Reinforcement Learning
• Model merging with SVD to tie the Knots
• Offline Reinforcement Learning with OOD State Correction and OOD Action Suppression
• Hokoff: Real Game Dataset from Honor of Kings and its Offline Reinforcement Learning Benchmarks
• The Llama 3 Herd of Models
• Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?
• HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning
• MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts
• Mixture of LoRA Experts
• GO4Align: Group Optimization for Multi-Task Alignment
• ReFT: Representation Finetuning for Language Models
• Asymmetry in Low-Rank Adapters of Foundation Models
• DoRA: Weight-Decomposed Low-Rank Adaptation
• Higher Layers Need More LoRA Experts
• LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin
• Sparse Low-rank Adaptation of Pre-trained Language Models
• Supported Trust Region Optimization for Offline Reinforcement Learning
• Parameter-Efficient Orthogonal Finetuning via Butterfly Factorization
• Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch
• Orthogonal Subspace Learning for Language Model Continual Learning
• Parameter Efficient Multi-task Model Fusion with Partial Linearization
• Counterfactual Conservative Q Learning for Offline Multi-agent Reinforcement Learning
• Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning
• LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuning
• LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition
• Controlling Text-to-Image Diffusion by Orthogonal Finetuning
• Soft Merging of Experts with Adaptive Routing
• TIES-Merging: Resolving Interference When Merging Models
• AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
• GPT-4 Technical Report
• AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Models
• Editing Models with Task Arithmetic
• Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering
• Bioinspired random projections for robust, sparse classification
• Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning
• Training language models to follow instructions with human feedback
• Federated Nearest Neighbor Classification with a Colony of Fruit-Flies: With Supplement
• Merging Models with Fisher-Weighted Averaging
• Training Verifiers to Solve Math Word Problems
• Evaluating Large Language Models Trained on Code
• BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
• Hash Layers For Large Sparse Models
• The Power of Scale for Parameter-Efficient Prompt Tuning
• GPT Understands, Too
• Can a Fruit Fly Learn Word Embeddings?
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning
• Measuring Massive Multitask Language Understanding
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Language Models are Few-Shot Learners
• Gradient Surgery for Multi-Task Learning
• Bio-Inspired Hashing for Unsupervised Similarity Search
• Similarity of Neural Network Representations Revisited
• Parameter-Efficient Transfer Learning for NLP
• Improving Similarity Search with High-dimensional Locality-sensitive Hashing
• Measuring the Intrinsic Dimension of Objective Landscapes
• Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• The Fast Johnson–Lindenstrauss Transform and Approximate Nearest Neighbors
• Adaptive Mixtures of Local Experts
• The Approximation of One Matrix by Another of Lower Rank
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• Fly-CL: A Fly-Inspired Framework for Enhancing Efficient Decorrelation and Reduced Training Time in Pre-trained Model-based Continual Representation Learning
• Utility-Diversity Aware Online Batch Selection for LLM Supervised Fine-tuning
• MoA: Heterogeneous Mixture of Adapters for Parameter-Efficient Fine-Tuning of Large Language Models

Related

• TT-LoRA MoE: Unifying Parameter-Efficient Fine-Tuning and Sparse Mixture-of-Experts
• MoE²-LoRA: When MoE Models Meet MoE-style Low-Rank Adaptation
• FURINA: Free from Unmergeable Router via LINear Aggregation of mixed experts
• MALoRA: Mixture of Asymmetric Low-Rank Adaptation for Enhanced Multi-Task Learning
• Each Rank Could be an Expert: Single-Ranked Mixture of Experts LoRA for Multi-Task Learning