DMA: Online RAG Alignment with Human Feedback

2025/11/06 by Bai, Yu, Miao, Yukai, Wang, Dawei +9
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Information Retrieval and Search Behavior #Recommender Systems and Techniques #Topic Modeling

paper · doi:10.48550/arxiv.2511.04880

openalex publication_date 2025/11/06 · openalex created_date 2025/11/11 · openalex updated_date 2026/07/28

Abstract

Retrieval-augmented generation (RAG) systems often rely on static retrieval, limiting adaptation to evolving intent and content drift. We introduce Dynamic Memory Alignment (DMA), an online learning framework that systematically incorporates multi-granularity human feedback to align ranking in interactive settings. DMA organizes document-, list-, and response-level signals into a coherent learning pipeline: supervised training for pointwise and listwise rankers, policy optimization driven by response-level preferences, and knowledge distillation into a lightweight scorer for low-latency serving. Throughout this paper, memory refers to the model's working memory, which is the entire context visible to the LLM for In-Context Learning. We adopt a dual-track evaluation protocol mirroring deployment: (i) large-scale online A/B ablations to isolate the utility of each feedback source, and (ii) few-shot offline tests on knowledge-intensive benchmarks. Online, a multi-month industrial deployment further shows substantial improvements in human engagement. Offline, DMA preserves competitive foundational retrieval while yielding notable gains on conversational QA (TriviaQA, HotpotQA). Taken together, these results position DMA as a principled approach to feedback-driven, real-time adaptation in RAG without sacrificing baseline capability.

Citations

• Qwen2 Technical Report
• Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity
• RAFT: Adapting Language Model to Domain Specific RAG
• Reliable, Adaptable, and Attributable Language Models with Retrieval
• REAR: A Relevance-Aware Retrieval-Augmented Framework for Open-Domain Question Answering
• Mixtral of Experts
• Learning to Filter Context for Retrieval-Augmented Generation
• Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answering
• Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewriting
• Making Retrieval-Augmented Language Models Robust to Irrelevant Context
• Efficient Memory Management for Large Language Model Serving with PagedAttention
• Lost in the Middle: How Language Models Use Long Contexts
• Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting
• Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy
• SAIL: Search-Augmented Instruction Learning
• Improving Language Models via Plug-and-Play Retrieval Feedback
• Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents
• How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrieval
• REPLUG: Retrieval-Augmented Black-Box Language Models
• Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
• Self-Instruct: Aligning Language Models with Self-Generated Instructions
• Re2G: Retrieve, Rerank, Generate

Related

• Pistis-RAG: Enhancing Retrieval-Augmented Generation with Human Feedback
• RPO: Retrieval Preference Optimization for Robust Retrieval-Augmented Generation
• Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generation
• Test-time Corpus Feedback: From Retrieval to RAG
• Optimizing Retrieval for RAG via Reinforced Contrastive Learning