LEDGERMIND: Provenance-Constrained Multimodal Agentic Reasoning with a Structured Evidence Ledger
2026/07/30 by Enjun Du, Hange Zhou, Chenxu Du +4
Computer Science · #cs.LG
paper · pdf
arxiv created 2026/07/30 · arxiv updated 2026/07/31
Abstract
Multimodal agents for visual question answering increasingly operate as multi-step trajectories that interleave perception, retrieval, and reasoning, yet evaluation still largely reduces to final-answer accuracy. This aggregate signal cannot tell whether a correct answer was reached through grounded evidence, language priors, or accidental error cancellation. We propose to treat a multimodal agent trajectory as a provenance-constrained state machine: tool outputs are normalized into a Structured Evidence Ledger that serves as the trajectory state, downstream reasoning and decision claims may cite only active ledger entries, grounding is checked at the entity and numeric level, and repair is realized as typed state transitions that cannot introduce content without tool-produced provenance. We instantiate this design as LedgerMind (Provenance-Constrained Multimodal Agentic Reasoning with a Structured Evidence Ledger), augmented by a Three-Layer Grounding Protocol, an Adaptive Dual-Path Dispatcher that matches reasoning depth to question complexity, and an Event-Triggered Verification-and-Repair engine with a formal provenance non-amplification guarantee. We use LedgerMind to target four recurring failure patterns that final-answer accuracy tends to obscure: unsupported intermediate reasoning, citation-backed entity hallucination (Phantom Grounding), over-reasoning on simple queries, and repair-time amplification. Experiments across multiple multimodal reasoning benchmarks and backbone MLLMs show that LedgerMind improves both answer accuracy and trajectory-level faithfulness.
Citations
- VideoSeg-R1:Reasoning Video Object Segmentation via Reinforcement Learning
- Octopus: Agentic Multimodal Reasoning with Six-Capability Orchestration
- TIR-Bench: A Comprehensive Benchmark for Agentic Thinking-with-Images Reasoning
- Decoupling Continual Semantic Segmentation
- WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agent
- SE-Agent: Self-Evolution Trajectory Optimization in Multi-Step Reasoning with LLM-Based Agents
- Don't Overthink it. Preferring Shorter Thinking Chains for Improved LLM Reasoning
- Between Underthinking and Overthinking: An Empirical Study of Reasoning Length and correctness in LLMs
- GraphMaster: Automated Graph Synthesis via LLM Agents in Data-Limited Environments
- Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models
- MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency
- Can MLLMs Reason in Multimodality? EMMA: An Enhanced MultiModal ReAsoning Benchmark
- Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs
- Correctness is not Faithfulness in RAG Attributions
- MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark
- Hallucination of Multimodal Large Language Models: A Survey
- Are We on the Right Way for Evaluating Large Vision-Language Models?
- Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinement
- On the Self-Verification Limitations of Large Language Models on Reasoning and Planning Tasks
- InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
- OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation
- MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI
- The Troubling Emergence of Hallucination in Large Language Models -- An Extensive Definition, Quantification, and Prescriptive Remediations
- Large Language Models Cannot Self-Correct Reasoning Yet
- Chain-of-Verification Reduces Hallucination in Large Language Models
- Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
- Measuring Faithfulness in Chain-of-Thought Reasoning
- Enabling Large Language Models to Generate Text with Citations
- Tree of Thoughts: Deliberate Problem Solving with Large Language Models
- Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting
- Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models
- Evaluating Verifiability in Generative Search Engines
- Visual Instruction Tuning
- Self-Refine: Iterative Refinement with Self-Feedback
- Reflexion: Language Agents with Verbal Reinforcement Learning
- GPT-4 Technical Report
- ViperGPT: Visual Inference via Python Execution for Reasoning
- Toolformer: Language Models Can Teach Themselves to Use Tools
- Faithful Chain-of-Thought Reasoning
- Distributing Accountability, Not Capability: Phase Separation and the LLM Workflow Quadrant in Autonomous AI Agent Architectures
- Self-Consistency Improves Chain of Thought Reasoning in Language Models
- BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
Related