Altogether: Image Captioning via Re-aligning Alt-text

2024/10/22 by Hu Xu, Po-Yao Huang, Xu, Hu +23 · 11 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Video Analysis and Summarization

paper · pdf · doi:10.48550/arxiv.2410.17251

openalex publication_date 2024/10/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

This paper focuses on creating synthetic data to improve the quality of image captions. Existing works typically have two shortcomings. First, they caption images from scratch, ignoring existing alt-text metadata, and second, lack transparency if the captioners' training data (e.g. GPT) is unknown. In this paper, we study a principled approach Altogether based on the key idea to edit and re-align existing alt-texts associated with the images. To generate training data, we perform human annotation where annotators start with the existing alt-text and re-align it to the image content in multiple rounds, consequently constructing captions with rich visual concepts. This differs from prior work that carries out human annotation as a one-time description task solely based on images and annotator knowledge. We train a captioner on this data that generalizes the process of re-aligning alt-texts at scale. Our results show our Altogether approach leads to richer image captions that also improve text-to-image generation and zero-shot image classification tasks.

Cited by

• Same or Not? Enhancing Visual Perception in Vision-Language Models
• Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning
• Caption Injection for Optimization in Generative Search Engine
• RoboSVG: A Unified Framework for Interactive SVG Generation with Multi-modal Guidance
• Probing the Limits of Stylistic Alignment in Vision-Language Models
• Brought a Gun to a Knife Fight: Modern VFM Baselines Outgun Specialized Detectors on In-the-Wild AI Image Detection
• ScaleCap: Inference-Time Scalable Image Captioning via Dual-Modality Debiasing
• Using Knowledge Graphs to harvest datasets for efficient CLIP model training
• RICO: Improving Accuracy and Completeness in Image Recaptioning via Visual Reconstruction
• EgoBabyVLM: Benchmarking Cross-Modal Learning from Naturalistic Egocentric Video Data
• On the Value of Cross-Modal Misalignment in Multimodal Representation Learning

Related

• Mining Fine-Grained Image-Text Alignment for Zero-Shot Captioning via Text-Only Training
• SynC: Synthetic Image Caption Dataset Refinement with One-to-many Mapping for Zero-shot Image Captioning
• CapText: Large Language Model-based Caption Generation From Image Context and Description
• Precision or Recall? An Analysis of Image Captions for Training Text-to-Image Generation Model
• Guiding Image Captioning Models Toward More Specific Captions