HERALD: Counterfactual Audits and Minimal Repairs for Proof-of-Retrieval Rewards

2026/08/06 by Zhuowen Liu, Bohan Cui, YinShang Guo +2
Computer Science · #cs.AI

paper · pdf

9 pages, 3 figures, and 4 tables

arxiv created 2026/08/06 · arxiv updated 2026/08/07

Abstract

Search-agent rewards mix answer quality, citation grounding, tool cost, and anti-hacking terms; a high score therefore need not imply that cited evidence was retrieved, and added penalties can cancel. We introduce HERALD, an offline audit that applies exact same-question interventions, separates candidate-visible from oracle information, and enumerates detector contracts before policy optimization. On four Qwen3-8B pools from HotpotQA, 2WikiMultiHopQA, and MuSiQue, R₀ rejects search deletion and fake IDs, but a label-free citation-laundering attack succeeds. A complete 2³ ablation identifies targeted strengthening of L---citing a corpus passage absent from the retrieved evidence---as the observed inclusion-minimal repair: R[L] has zero empirical ASR with a 0.50% one-sided cluster upper bound. The gap persists across pool rules, a visible BM25 attacker, and four models; broader hardening remains vulnerable when the attack removes an oracle support-ID penalty. Under strict 5M-token matched training evaluated on 256 paired questions per benchmark, R[L] meets the EM non-inferiority gate on HotpotQA and 2Wiki but not MuSiQue. Equal-suite citation precision and support recall improve by 2.02 and 1.46 points, unsupported citations fall by 1.69, and laundering attackability falls on 2Wiki and MuSiQue. Natural L is not reduced, and the detector appears in only 18 of 58,368 training trajectories. HERALD thus separates robust scoring, sparse learning signal, and policy transfer.

Citations

• Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use
• LLMs Gaming Verifiers: RLVR can Lead to Reward Hacking
• Proof-of-Use: Mitigating Tool-Call Hacking in Deep Research Agents
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers
• DynaSearcher: Dynamic Knowledge Graph Augmented Search Agent via Multi-Reward Reinforcement Learning
• One Token to Fool LLM-as-a-Judge
• Spurious Rewards: Rethinking Training Signals in RLVR
• A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility
• Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
• RATE: Causal Explainability of Reward Models with Imperfect Counterfactuals
• Enabling Large Language Models to Generate Text with Citations
• Evaluating Verifiability in Generative Search Engines
• Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
• Scaling Laws for Reward Model Overoptimization
• Distributing Accountability, Not Capability: Phase Separation and the LLM Workflow Quadrant in Autonomous AI Agent Architectures
• Measuring Attribution in Natural Language Generation Models
• Measuring Attribution in Natural Language Generation Models
• MuSiQue: Multihop Questions via Single-hop Question Composition
• Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Beyond Accuracy: Behavioral Testing of NLP models with CheckList
• Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective
• HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
• Reinforcement Learning with a Corrupted Reward Channel