Geometric Risk Control for Vision-Language Model OCR

2026/03/31 by Weile Gong, Zijian Lu, Mingcai Chen +3
Computer Science · #cs.CV

paper · pdf

9 pages, 5 figures, Code: https://github.com/phare111/GRC

arxiv created 2026/07/30 · arxiv updated 2026/07/31

Abstract

Vision-language models (VLMs) enable flexible generative optical character recognition (OCR), while their open-ended decoders can expose wrong but fluent text with weak visual support. In audit-sensitive records, such an output can be more costly than abstention. Frozen or externally served VLMs therefore require an external decision layer that can determine whether a transcription has sufficient visual evidence for release. We introduce the Geometric Risk Controller (GRC), a model-agnostic controller that treats controlled geometric transformations as repeatable black-box probes, screens structurally implausible continuations, and releases the unique candidate supported by coherent cross-view evidence. The protocol provides empirical selective exposure control with explicit coverage and query cost under a reproducible fixed decision rule. Experiments across frozen VLMs and standard scene-text benchmarks consistently reduce mean, upper-tail, and catastrophic error among released outputs while retaining high coverage.

Citations

• SAFER: Risk-Constrained Sample-then-Filter in Large Language Models
• Efficient Test-Time Scaling for Small Vision-Language Models
• PARC: A Quantitative Framework Uncovering the Symmetries within Vision Language Models
• MTRE: Multi-Token Reliability Estimation for Hallucination Detection in VLMs
• Ocean-OCR: Towards General OCR Application via a Vision-Language Model
• Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters
• Know Your Limits: A Survey of Abstention in Large Language Models
• Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Models
• TextHawk: Exploring Efficient Fine-Grained Perception of Multimodal Large Language Models
• Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoning
• LLaVA-Phi: Efficient Multi-Modal Assistant with Small Language Model
• OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation
• Efficient Guided Generation for Large Language Models
• Evaluating Object Hallucination in Large Vision-Language Models
• Conformal Risk Control
• Self-Consistency Improves Chain of Thought Reasoning in Language Models
• A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification
• Calibrate Before Use: Improving Few-Shot Performance of Language Models
• Selective Classification for Deep Neural Networks

Related

• Reading or Guessing? Visual Grounding Failures of Vision-Language Models for OCR in Ancient Greek Editions
• DianJin-OCR-R1: Enhancing OCR Capabilities via a Reasoning-and-Tool Interleaved Vision-Language Model
• Towards Statistical Factuality Guarantee for Large Vision-Language Models
• Do VLMs Read or Rewrite? On Transcription Faithfulness in Vision-Language Models
• Layer-wise Alignment: Examining Safety Alignment Across Image Encoder Layers in Vision Language Models