Benchmarking Vision-Language Models on Optical Character Recognition in Dynamic Video Environments
2025/02/10 by Sankalp Nagaonkar, Nagaonkar, Sankalp, Augustya Sharma +5 · 11 voices · 5 citations
Computer Science · Earth and Planetary Sciences · #Handwritten Text Recognition Techniques #Advanced Image and Video Retrieval Techniques #3D Surveying and Cultural Heritage
paper · pdf · doi:10.48550/arxiv.2502.06445
Abstract
This paper introduces an open-source benchmark for evaluating Vision-Language Models (VLMs) on Optical Character Recognition (OCR) tasks in dynamic video environments. We present a curated dataset containing 1,477 manually annotated frames spanning diverse domains, including code editors, news broadcasts, YouTube videos, and advertisements. Three state of the art VLMs - Claude-3, Gemini-1.5, and GPT-4o are benchmarked against traditional OCR systems such as EasyOCR and RapidOCR. Evaluation metrics include Word Error Rate (WER), Character Error Rate (CER), and Accuracy. Our results highlight the strengths and limitations of VLMs in video-based OCR tasks, demonstrating their potential to outperform conventional OCR models in many scenarios. However, challenges such as hallucinations, content security policies, and sensitivity to occluded or stylized text remain. The dataset and benchmarking framework are publicly available to foster further research.
Cited by
Discussions
- Benchmarking vision-language models on OCR in dynamic video environments [hn, 142 points, 58 comments]
- Gemini beats everyone on new OCR benchmark [bsky, 0 points, 0 comments]
- Gemini beats everyone on new OCR benchmark #HackerNews arxiv.org/abs/... [bsky, 0 points, 0 comments]
- Benchmarking Vision-Language Models on OCR in Dynamic Video Environments https://arxiv.org/abs/2502.06445 https://news.ycombinator.com/item?id=43045801 [bsky, 0 points, 0 comments]
- Gemini beats everyone on new OCR benchmark https://arxiv.org/abs/2502.06445 [bsky, 0 points, 0 comments]
- Gemini beats everyone on new OCR benchmark https://arxiv.org/abs/2502.06445 [comments] [79 points] [bsky, 0 points, 0 comments]
- Gemini beats everyone on new OCR benchmark https://arxiv.org/abs/2502.06445 (https://news.ycombinator.com/item?id=43045801) [bsky, 0 points, 0 comments]
- https://bsky.app/profile/news.ycombinator.com.web.brid.gy/post/3li5bh2zvo462 [bsky, 0 points, 0 comments]
- Benchmarking vision-language models on OCR in dynamic video environments (arxiv.org) Main Link | Discussion [bsky, 0 points, 0 comments]
- Gemini beats everyone on new OCR benchmark view on hacker news [bsky, 0 points, 0 comments]
- Gemini beats everyone on new OCR benchmark https://arxiv.org/abs/2502.06445 (https://news.ycombinator.com/item?id=43045801) [bsky, 0 points, 0 comments]
Related