LOTUS: A Leaderboard for Detailed Image Captioning from Quality to Societal Bias and User Preferences

2025/07/25 by Yusuke Hirota, Boyi Li, Hirota, Yusuke +17 · 1 citation
Arts and Humanities · Computer Science · #Closed captioning #Multimodal Machine Learning Applications #Preference #Preference elicitation #Quality (philosophy) #Selection (genetic algorithm) #Subtitles and Audiovisual Media #User modeling #Visual Attention and Saliency Detection

paper · open access · doi:10.48550/arxiv.2507.19362

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/07/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Vision-Language Models (LVLMs) have transformed image captioning, shifting from concise captions to detailed descriptions. We introduce LOTUS, a leaderboard for evaluating detailed captions, addressing three main gaps in existing evaluations: lack of standardized criteria, bias-aware assessments, and user preference considerations. LOTUS comprehensively evaluates various aspects, including caption quality (e.g., alignment, descriptiveness), risks (\eg, hallucination), and societal biases (e.g., gender bias) while enabling preference-oriented evaluations by tailoring criteria to diverse user preferences. Our analysis of recent LVLMs reveals no single model excels across all criteria, while correlations emerge between caption detail and bias risks. Preference-oriented evaluations demonstrate that optimal model selection depends on user priorities.

Citations

• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• SANER: Annotation-free Societal Attribute Neutralizer for Debiasing CLIP
• Resampled Datasets Are Not Enough: Mitigating Societal Bias Beyond Single Attributes
• DOCCI: Descriptions of Connected and Contrasting Images
• DreamLIP: Language-Image Pre-training with Long Captions
• Long-CLIP: Unlocking the Long-Text Capability of CLIP
• MLLMs-Augmented Visual-Language Representation Learning
• OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation
• Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding
• ShareGPT4V: Improving Large Multi-Modal Models with Better Captions
• mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration
• VeCLIP: Improving CLIP Training via Visual-enriched Captions
• Improved Baselines with Visual Instruction Tuning
• InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning
• Can Large Language Models Be an Alternative to Human Evaluations?
• Model-Agnostic Gender Debiased Image Captioning
• Uncurated Image-Text Datasets: Shedding Light on Demographic Bias
• IC3: Image Captioning by Committee Consensus
• Men Also Do Laundry: Multi-Attribute Bias Amplification
• OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework
• Understanding and Evaluating Racial Biases in Image Captioning
• Learning Transferable Visual Models From Natural Language Supervision
• Mitigating Gender Bias in Captioning Systems
• Connecting Vision and Language with Localized Narratives
• Women also Snowboard: Overcoming Bias in Captioning Models
• Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Deep Visual-Semantic Alignments for Generating Image Descriptions
• CIDEr: Consensus-based Image Description Evaluation

Cited by

• HalDec-Bench: Benchmarking Hallucination Detector in Image Captioning

Related

• Painting with Words: Elevating Detailed Image Captioning with Benchmark and Alignment Learning
• CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Era
• Surveying the Landscape of Image Captioning Evaluation: A Comprehensive Taxonomy, Trends and Metrics Analysis
• Image Captioning Evaluation in the Age of Multimodal LLMs: Challenges and Future Perspectives
• Transparent Human Evaluation for Image Captioning