SEQR: Secure and Efficient QR-based LoRA Routing

2025/09/22 by William Fleshman, Benjamin Van Durme, Fleshman, William +1
Biochemistry, Genetics and Molecular Biology · Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Cooperative Communication and Network Coding #DNA and Biological Computing #Energy Efficient Wireless Sensor Networks #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2509.18093

openalex publication_date 2025/09/22 · openalex created_date 2025/10/16 · openalex updated_date 2026/07/28

Abstract

Low-Rank Adaptation (LoRA) has become a standard technique for parameter-efficient fine-tuning of large language models, enabling large libraries of LoRAs, each for a specific task or domain. Efficiently selecting the correct LoRA adapter for a given input remains a challenge, particularly in secure environments where supervised training of routers may raise privacy concerns. Motivated by previous approaches, we formalize the goal of unsupervised LoRA routing in terms of activation norm maximization, providing a theoretical framework for analysis. We demonstrate the discriminative power of activation norms and introduce SEQR, an unsupervised LoRA routing algorithm designed to maximize efficiency while providing strict routing guarantees. SEQR provably identifies the norm-maximizing adapter with significantly greater efficiency, making it a highly scalable and effective solution for dynamic LoRA composition. We validate our results through experiments that demonstrate improved multi-task performance and efficiency.

Citations

• LoRA-Augmented Generation (LAG) for Knowledge-Intensive Language Tasks
• SpectR: Dynamically Composing LM Experts with Spectral Routing
• LessLeak-Bench: A First Investigation of Data Leakage in LLMs Across 83 Software Engineering Benchmarks
• Model merging with SVD to tie the Knots
• Selective Aggregation for Low-Rank Adaptation in Federated Learning
• Risks When Sharing LoRA Fine-Tuned Diffusion Model Weights
• Representation Norm Amplification for Out-of-Distribution Detection in Long-Tail Learning
• Towards Scalable Exact Machine Unlearning Using Parameter-Efficient Fine-Tuning
• Compress then Serve: Serving Thousands of LoRA Adapters with Little Overhead
• Towards Modular LLMs by Building and Reusing a Library of LoRAs
• AdapterSwap: Continuous Training of LLMs with Data Removal and Access-Control Guarantees
• Improving LoRA in Privacy-preserving Federated Learning
• Out-of-Distribution Detection using Neural Activation Prior
• Massive Activations in Large Language Models
• Asymmetry in Low-Rank Adapters of Foundation Models
• Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch
• Parameter Efficient Multi-task Model Fusion with Partial Linearization
• Understanding the Feature Norm for Out-of-Distribution Detection
• Pushing Mixture of Experts to the Limit: Extremely Parameter Efficient MoE for Instruction Tuning
• LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Composition
• Neuron Activation Coverage: Rethinking Out-of-distribution Detection and Generalization
• TIES-Merging: Resolving Interference When Merging Models
• AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Models
• Editing Models with Task Arithmetic
• Multi-Head Adapter Routing for Cross-Task Generalization
• AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning
• Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time
• Membership Inference Attacks From First Principles
• LoRA: Low-Rank Adaptation of Large Language Models
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• HuggingFace's Transformers: State-of-the-art Natural Language Processing
• HellaSwag: Can a Machine Really Finish Your Sentence?
• Neural Network Acceptability Judgments
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference
• Membership Inference Attacks against Machine Learning Models
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Adaptive Mixtures of Local Experts

Related

• HiLoRA: Adaptive Hierarchical LoRA Routing for Training-Free Domain Generalization
• LoRA-Mixer: Coordinate Modular LoRA Experts Through Serial Attention Routing
• MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning
• Learning to Select, Not Relearn: Hard-Routed Mixtures of Reasoning LoRAs
• LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Experts