Noise-Robustness Through Noise: A Framework combining Asymmetric LoRA with Poisoning MoE

2025/05/29 by Wang, Zhaokun, Guo, Jinyu, Pu, Jingwen +5 · 1 citation
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2505.23868

Abstract

Current parameter-efficient fine-tuning methods for adapting pre-trained language models to downstream tasks are susceptible to interference from noisy data. Conventional noise-handling approaches either rely on laborious data pre-processing or employ model architecture modifications prone to error accumulation. In contrast to existing noise-process paradigms, we propose a noise-robust adaptation method via asymmetric LoRA poisoning experts (LoPE), a novel framework that enhances model robustness to noise only with generated noisy data. Drawing inspiration from the mixture-of-experts architecture, LoPE strategically integrates a dedicated poisoning expert in an asymmetric LoRA configuration. Through a two-stage paradigm, LoPE performs noise injection on the poisoning expert during fine-tuning to enhance its noise discrimination and processing ability. During inference, we selectively mask the dedicated poisoning expert to leverage purified knowledge acquired by normal experts for noise-robust output. Extensive experiments demonstrate that LoPE achieves strong performance and robustness purely through the low-cost noise injection, which completely eliminates the requirement of data cleaning.

Citations

• HASH-RAG: Bridging Deep Hashing with Retriever for Efficient, Fine Retrieval and Augmented Generation
• Accelerating Adaptive Retrieval Augmented Generation via Instruction-Driven Representation Reduction of Retrieval Overlaps
• Hide and Seek in Noise Labels: Noise-Robust Collaborative Active Learning with LLM-Powered Assistance
• MOOSE-Chem: Large Language Models for Rediscovering Unseen Chemistry Scientific Hypotheses
• Fine-Tuning is Fine, if Calibrated
• Concentrate Attention: Towards Domain-Generalizable Prompt Optimization for Language Models
• An Information Bottleneck Perspective for Effective Noise Filtering on Retrieval-Augmented Generation
• HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning
• LLMClean: Context-Aware Tabular Data Cleaning via LLM-Generated OFDs
• MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts
• Advancing the Robustness of Large Language Models through Self-Denoised Smoothing
• Noise-BERT: A Unified Perturbation-Robust Framework with Noise Alignment Pre-training for Noisy Slot Filling Task
• How Well Can LLMs Negotiate? NegotiationArena Platform and Analysis
• Fine tuning Pre trained Models for Robustness Under Noisy Labels
• MProto: Multi-Prototype Network with Denoised Optimal Transport for Distantly Supervised Named Entity Recognition
• NEFTune: Noisy Embeddings Improve Instruction Finetuning
• Qwen Technical Report
• Llama 2: Open Foundation and Fine-Tuned Chat Models
• Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learner
• ASSIST: Towards Label Noise-Robust Dialogue State Tracking
• Training Verifiers to Solve Math Word Problems
• GPT Understands, Too
• Measuring Massive Multitask Language Understanding
• Learning from Noisy Labels with Deep Neural Networks: A Survey
• PIQA: Reasoning about Physical Commonsense in Natural Language
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• Prefix-Tuning: Optimizing Continuous Prompts for Generation
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• MANTRA: a Framework for Multi-stage Adaptive Noise TReAtment During Training

Related

• Noise Augmented Fine Tuning for Mitigating Hallucinations in Large Language Models
• Weed Out, Then Harvest: Dual Low-Rank Adaptation is an Effective Noisy Label Detector for Noise-Robust Learning
• LoRA-Leak: Membership Inference Attacks Against LoRA Fine-tuned Language Models
• BA-LoRA: Bias-Alleviating Low-Rank Adaptation to Mitigate Catastrophic Inheritance in Large Language Models
• RobustFT: Robust Supervised Fine-tuning for Large Language Models under Noisy Response