Beyond Memorization: A Multi-Modal Ordinal Regression Benchmark to Expose Popularity Bias in Vision-Language Models

2025/12/24 by Li-Zhong Szu-Tu, Szu-Tu, Li-Zhong, Ting-Lin Wu +7
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Sentiment Analysis and Opinion Mining

paper · doi:10.48550/arxiv.2512.21337

openalex publication_date 2025/12/24 · openalex created_date 2025/12/26 · openalex updated_date 2026/07/28

Abstract

We expose a significant popularity bias in state-of-the-art vision-language models (VLMs), which achieve up to 34% higher accuracy on famous buildings compared to ordinary ones, indicating a reliance on memorization over generalizable understanding. To systematically investigate this, we introduce the largest open benchmark for this task: the YearGuessr dataset, a collection of 55,546 building images with multi-modal attributes from 157 countries, annotated with continuous ordinal labels of their construction year (1001-2024), GPS data, and page-view counts as a proxy for popularity. Using this dataset, we frame the construction year prediction task as ordinal regression and introduce popularity-aware interval accuracy metrics to quantify this bias. Our resulting benchmark of 30+ models, including our YearCLIP model, confirms that VLMs excel on popular, memorized items but struggle significantly with unrecognized subjects, exposing a critical flaw in their reasoning capabilities. Project page: https://sytwu.github.io/BeyondMemo/

Citations

• Skyfall-GS: Synthesizing Immersive 3D Urban Scenes from Satellite Imagery
• See, Point, Fly: A Learning-Free VLM Framework for Universal Unmanned Aerial Navigation
• OpenM3D: Open Vocabulary Multi-view Indoor 3D Object Detection without Human Annotations
• Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs
• Qwen2.5-VL Technical Report
• Building Age Estimation: A New Multi-Modal Benchmark Dataset and Community Challenge
• CMAB: A First National-Scale Multi-Attribute Building Dataset in China Derived from Open Source Data and GeoAI
• Teach CLIP to Develop a Number Sense for Ordinal Regression
• MiniCPM-V: A GPT-4V Level MLLM on Your Phone
• AddressCLIP: Empowering Vision-Language Models for City-wide Image Address Localization
• Depth Anywhere: Enhancing 360 Monocular Depth Estimation via Perspective Distillation and Unlabeled Data Augmentation
• LLMGeo: Benchmarking Large Language Models on Image Geolocation In-the-wild
• Zero-shot Building Age Classification from Facade Image Using GPT-4
• Image-Text Co-Decomposition for Text-Supervised Semantic Segmentation
• InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
• Improved Baselines with Visual Instruction Tuning
• Predicting building age from urban form at large scale
• GeoCLIP: Clip-Inspired Alignment between Locations and Images for Effective Worldwide Geo-localization
• Can Vision-Language Models be a Good Guesser? Exploring VLMs for Times and Location Reasoning
• PIGEON: Predicting Image Geolocations
• SNAP: Self-Supervised Neural Maps for Visual Positioning and Semantic Understanding
• CrowdCLIP: Unsupervised Crowd Counting via Vision-Language Model
• GPT-4 Technical Report
• Where We Are and What We're Looking At: Query Based Worldwide Image Geo-localization Using Hierarchies and Scenes
• EXIF as Language: Learning Cross-Modal Associations Between Images and Camera Metadata
• SatlasPretrain: A Large-Scale Dataset for Remote Sensing Image Understanding
• G³: Geolocation via Guidebook Grounding
• OrdinalCLIP: Learning Rank Prompts for Language-Guided Ordinal Regression
• Where in the World is this Image? Transformer-based Geo-localization in the Wild
• TransGeo: Transformer Is All You Need for Cross-view Image Geo-localization
• Moving Window Regression: A Novel Approach to Ordinal Regression
• A ConvNet for the 2020s
• Learning Probabilistic Ordinal Embeddings for Uncertainty-Aware Regression
• Learning Transferable Visual Models From Natural Language Supervision
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets
• Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation
• Rank consistent ordinal regression for neural networks with application to age estimation
• Predicting residential building age from map data
• CPlaNet: Enhancing Image Geolocalization by Combinatorial Partitioning of Maps
• Deep Ordinal Regression Network for Monocular Depth Estimation
• No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing World
• Revisiting IM2GPS in the Deep Learning Era
• Deep Residual Learning for Image Recognition
• Material Recognition in the Wild with the Materials in Context Database
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows

Related

• STORM: Benchmarking Visual Rating of MLLMs with a Comprehensive Ordinal Regression Dataset
• Vision Language Models are Biased
• Teach CLIP to Develop a Number Sense for Ordinal Regression
• AI vs Human Expert Reasoning: Assessing Agreements in Building Typology Predictions based on Street View Imagery
• Scaling Vision-Language Models Is Not Enough to Mitigate Bias