Can Current AI Models Count What We Mean, Not What They See? A Benchmark and Systematic Evaluation

2025/09/17 by Nguyen, Gia Khanh, Huang, Yifeng, Hoai, Minh · 1 citation
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2509.13939

Abstract

Visual counting is a fundamental yet challenging task, especially when users need to count objects of a specific type in complex scenes. While recent models, including class-agnostic counting models and large vision-language models (VLMs), show promise in counting tasks, their ability to perform fine-grained, intent-driven counting remains unclear. In this paper, we introduce PairTally, a benchmark dataset specifically designed to evaluate fine-grained visual counting. Each of the 681 high-resolution images in PairTally contains two object categories, requiring models to distinguish and count based on subtle differences in shape, size, color, or semantics. The dataset includes both inter-category (distinct categories) and intra-category (closely related subcategories) settings, making it suitable for rigorous evaluation of selective counting capabilities. We benchmark a variety of state-of-the-art models, including exemplar-based methods, language-prompted models, and large VLMs. Our results show that despite recent advances, current models struggle to reliably count what users intend, especially in fine-grained and visually ambiguous cases. PairTally provides a new foundation for diagnosing and improving fine-grained visual counting systems.

Citations

• InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models
• Qwen2.5-VL Technical Report
• A Survey on Class-Agnostic Counting: Advancements from Reference-Based to Open-World Text-Guided Approaches
• LLMDet: Learning Strong Open-Vocabulary Object Detectors under the Supervision of Large Language Models
• A Novel Unified Architecture for Low-Shot Counting by Detection and Segmentation
• Mind the Prompt: A Novel Benchmark for Prompt-based Class-Agnostic Counting
• The Llama 3 Herd of Models
• Zero-shot Object Counting with Good Exemplars
• CountGD: Multi-Modal Open-World Counting
• Ovis: Structural Embedding Alignment for Multimodal Large Language Model
• DAVE -- A Detect-and-Verify Paradigm for Low-Shot Counting
• A Fixed-Point Approach to Unified Prompt-Based Counting
• OmniCount: Multi-label Object Counting with Semantic-Geometric Priors
• VLCounter: Text-aware Visual Representation for Zero-Shot Object Counting
• Point, Segment and Count: A Generalized Framework for Object Counting
• Interactive Class-Agnostic Object Counting
• ABC Easy as 123: A Blind Counter for Exemplar-Free Multi-Class Class-agnostic Counting
• Vision Transformer Off-the-Shelf: A Surprising Baseline for Few-Shot Class-Agnostic Counting
• Annotated Point Clouds and Images from a Spatial-Semantic Perception Pipeline: Robotized Deconstruction at the Reference Construction Site, Aachen
• Zero-shot Object Counting
• Teaching CLIP to Count to Ten
• A Low-Shot Object Counting Network With Iterative Prototype Adaptation
• Few-shot Object Counting and Detection
• Exemplar Free Class Agnostic Counting
• Represent, Compare, and Learn: A Similarity-Aware Framework for Class-Agnostic Counting
• Few-shot Object Counting with Similarity-Aware Feature Enhancement
• Rethinking Counting and Localization in Crowds:A Purely Point-Based Framework
• Learning To Count Everything
• Localization in the Crowd with Topological Constraints
• Uncertainty Estimation and Sample Selection for Crowd Counting
• Distribution Matching for Crowd Counting
• Adaptive Mixture Regression Network with Local Counting Map for Crowd Counting
• JHU-CROWD++: Large-Scale Crowd Counting Dataset and A Benchmark Method
• Shallow Feature Based Dense Attention Network for Crowd Counting
• NAS-Count: Counting-by-Density with Neural Architecture Search
• Learn to Scale: Generating Multipolar Normalized Density Maps for Crowd Counting
• Locate, Size and Count: Accurately Resolving People in Dense Crowds via Detection
• Locate, Size and Count: Accurately Resolving People in Dense Crowds via Detection
• Point in, Box out: Beyond Counting Persons in Crowds
• Class-Agnostic Counting
• Iterative Crowd Counting
• Where are the Blobs: Counting by Localization with Point Supervision
• CSRNet: Dilated Convolutional Neural Networks for Understanding the Highly Congested Scenes
• Drone-based Object Counting by Spatially Regularized Regional Proposal Network
• Open-world Text-specified Object Counting

Cited by

• CountGD++: Generalized Prompting for Open-World Counting

Related

• Assessing the Visual Enumeration Abilities of Specialized Counting Architectures and Vision-Language Models
• Your Vision-Language Model Can't Even Count to 20: Exposing the Failures of VLMs in Compositional Counting
• Can Vision-Language Models Count? A Synthetic Benchmark and Analysis of Attention-Based Interventions
• Visual Enumeration Remains Challenging for Multimodal Generative AI
• LVLM-COUNT: Enhancing the Counting Ability of Large Vision-Language Models