Annotation-Free Reinforcement Learning Query Rewriting via Verifiable Search Reward

2025/07/31 by Sungguk Cha, Cha, Sungguk, DongWook Kim +8 · 1 citation
Computer Science · #Web Data Mining and Analysis #Advanced Image and Video Retrieval Techniques #Information Retrieval and Search Behavior

paper · pdf · doi:10.48550/arxiv.2507.23242

Abstract

Optimizing queries for Retrieval-Augmented Generation (RAG) systems poses a significant challenge, particularly across diverse modal indices. We introduce RL-QR, a novel annotation-free reinforcement learning framework for query rewriting that eliminates the need for costly human-annotated data. By leveraging verifiable search rewards derived from index-aligned synthetic queries, RL-QR overcomes human-annotation dependencies, extending its applicability to various modalities and index domains. Experimental results demonstrate the framework's robustness, achieving substantial retrieval performance gains of up to 3.9× on lexical retrievers and 3.5× on semantic retrievers on the MTEB VIDORE V2 benchmark for unstructured visual documents, along with consistent 5% to 10% improvements on MS MARCO v2.1 and internal industrial datasets.

Citations

• Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models
• ViDoRe Benchmark V2: Raising the Bar for Visual Retrieval
• Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
• DMQR-RAG: Diverse Multi-Query Rewriting for RAG
• Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing
• From Local to Global: A Graph RAG Approach to Query-Focused Summarization
• RQ-RAG: Learning to Refine Queries for Retrieval Augmented Generation
• Atlas: Few-shot Learning with Retrieval Augmented Language Models
• Conversational Query Rewriting with Self-supervised Learning
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Dense Passage Retrieval for Open-Domain Question Answering
• MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
• Cumulated gain-based evaluation of IR techniques

Cited by

• Think Before You Retrieve: Learning Test-Time Adaptive Search with Small Language Models

Related

• Rethinking On-policy Optimization for Query Augmentation
• RaFe: Ranking Feedback Improves Query Rewriting for RAG
• DeepRetrieval: Hacking Real Search Engines and Retrievers with Large Language Models via Reinforcement Learning
• s3: You Don't Need That Much Data to Train a Search Agent via RL
• Harnessing the Power of Reinforcement Learning for Language-Model-Based Information Retriever via Query-Document Co-Augmentation