Towards Fine-Grained Recognition with Large Visual Language Models: Benchmark and Optimization Strategies

2025/12/11 by Cong Pang, Pang, Cong, Hongtao Yu +6
Computer Science · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2512.10384

openalex publication_date 2025/12/11 · openalex created_date 2025/12/13 · openalex updated_date 2026/07/28

Abstract

Large Vision Language Models (LVLMs) have made remarkable progress, enabling sophisticated vision-language interaction and dialogue applications. However, existing benchmarks primarily focus on reasoning tasks, often neglecting fine-grained recognition, which is crucial for practical application scenarios. To address this gap, we introduce the Fine-grained Recognition Open World (FROW) benchmark, designed for detailed evaluation of LVLMs with GPT-4o. On the basis of that, we propose a novel optimization strategy from two perspectives: data construction and training process, to improve the performance of LVLMs. Our dataset includes mosaic data, which combines multiple short-answer responses, and open-world data, generated from real-world questions and answers using GPT-4o, creating a comprehensive framework for evaluating fine-grained recognition in LVLMs. Experiments show that mosaic data improves category recognition accuracy by 1% and open-world data boosts FROW benchmark accuracy by 10%-20% and content accuracy by 6%-12%. Meanwhile, incorporating fine-grained data into the pre-training phase can improve the model's category recognition accuracy by up to 10%. The benchmark will be available at https://github.com/pc-inno/FROW.

Citations

• Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencent
• GPT-4o System Card
• African or European Swallow? Benchmarking Large Vision-Language Models for Fine-Grained Object Classification
• ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Tools
• Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models
• Why are Visually-Grounded Language Models Bad at Image Classification?
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
• InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
• MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts
• Q-Bench: A Benchmark for General-Purpose Foundation Models on Low-level Vision
• Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond
• SuperCLUE: A Comprehensive Chinese Large Language Model Benchmark
• Evaluating Object Hallucination in Large Vision-Language Models
• Visual Instruction Tuning
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
• ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning
• InfographicVQA
• DocVQA: A Dataset for VQA on Document Images
• AutoAugment: Learning Augmentation Policies from Data
• A Diagram Is Worth A Dozen Images
• Fine-Grained Visual Classification of Aircraft
• Needle In A Video Haystack: A Scalable Synthetic Evaluator for Video MLLMs

Related

• Benchmarking Large Vision-Language Models on Fine-Grained Image Tasks: A Comprehensive Evaluation
• Unveiling the Tapestry of Consistency in Large Vision-Language Models
• LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models
• Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Models
• ALLaVA: Harnessing GPT4V-Synthesized Data for Lite Vision-Language Models