GIT: A Generative Image-to-text Transformer for Vision and Language

2022/05/27 by Jianfeng Wang, Wang, Jianfeng, Zhengyuan Yang +15 · 2 voices · 47 citations
Computer Science · #Handwritten Text Recognition Techniques #Multimodal Machine Learning Applications #Natural Language Processing Techniques #cs.CV

paper · pdf · doi:10.48550/arxiv.2205.14100

openalex publication_date 2022/05/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this paper, we design and train a Generative Image-to-text Transformer, GIT, to unify vision-language tasks such as image/video captioning and question answering. While generative models provide a consistent network architecture between pre-training and fine-tuning, existing work typically contains complex structures (uni/multi-modal encoder/decoder) and depends on external modules such as object detectors/taggers and optical character recognition (OCR). In GIT, we simplify the architecture as one image encoder and one text decoder under a single language modeling task. We also scale up the pre-training data and the model size to boost the model performance. Without bells and whistles, our GIT establishes new state of the arts on 12 challenging benchmarks with a large margin. For instance, our model surpasses the human performance for the first time on TextCaps (138.2 vs. 125.5 in CIDEr). Furthermore, we present a new scheme of generation-based image classification and scene text recognition, achieving decent performance on standard benchmarks. Codes are released at \urlhttps://github.com/microsoft/GenerativeImage2Text.

Cited by

• UltraViT: Latency-Optimized On-device Vision Encoder for Large Vision-Language Models
• LLM-Free Image Captioning Evaluation in Reference-Flexible Settings
• brat: Aligned Multi-View Embeddings for Brain MRI Analysis
• MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentation
• Diffusion Is Your Friend in Show, Suggest and Tell
• Beyond Hallucinations: A Multimodal-Guided Task-Aware Generative Image Compression for Ultra-Low Bitrate
• SFA: Scan, Focus, and Amplify toward Guidance-aware Answering for Video TextVQA
• Dual-Granularity Semantic Prompting for Language Guidance Infrared Small Target Detection
• Thinking-while-Generating: Interleaving Textual Reasoning throughout Visual Generation
• SpaceVLM: Sub-Space Modeling of Negation in Vision-Language Models
• Revisiting the Data Sampling in Multimodal Post-training from a Difficulty-Distinguish View
• Masked Diffusion Captioning for Visual Feature Learning
• Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders
• Top-Down Semantic Refinement for Image Captioning
• CaptionFormer: Unified Segmentation, Tracking, and Captioning for Spatio-Temporal Objects
• Seeing and Knowing in the Wild: Open-domain Visual Entity Recognition with Large-scale Knowledge Graphs via Contrastive Learning
• Looking to Learn: Token-wise Dynamic Gating for Low-Resource Vision-Language Modelling
• Improving Chain-of-Thought Efficiency for Autoregressive Image Generation
• Model Merging to Maintain Language-Only Performance in Developmentally Plausible Multimodal Models
• What Drives Compositional Generalization in Visual Generative Models?
• Brain-Language Model Alignment: Insights into the Platonic Hypothesis and Intermediate-Layer Advantage
• VELA: An LLM-Hybrid-as-a-Judge Approach for Evaluating Long Image Captions
• IRIS: Intrinsic Reward Image Synthesis
• STAGE: Stable and Generalizable GRPO for Autoregressive Image Generation
• AutoPrune: Each Complexity Deserves a Pruning Policy
• Group Critical-token Policy Optimization for Autoregressive Image Generation
• MILR: Improving Multimodal Image Generation via Test-Time Latent Reasoning
• Global Minimizers of Sigmoid Contrastive Loss
• Multimodal Representation Learning Conditioned on Semantic Relations
• Towards Understanding Visual Grounding in Visual Language Models
• GeneVA: A Dataset of Human Annotations for Generative Text to Video Artifacts
• Rethinking the long-range dependency in Mamba/SSM and transformer models
• Time-Scaling State-Space Models for Dense Video Captioning
• OpenVision 2: A Family of Generative Pretrained Visual Encoders for Multimodal Learning
• EVENT-Retriever: Event-Aware Multimodal Image Retrieval for Realistic Captions
• Safe-LLaVA: A Privacy-Preserving Vision-Language Dataset and Benchmark for Biometric Safety
• VoCap: Video Object Captioning and Segmentation from Any Prompt
• MM-SeR: Multimodal Self-Refinement for Lightweight Image Captioning
• Looking Beyond the Obvious: A Survey on Abstract Concept Recognition for Video Understanding
• Event-Enriched Image Analysis Grand Challenge at ACM Multimedia 2025
• From Prediction to Explanation: Multimodal, Explainable, and Interactive Deepfake Detection Framework for Non-Expert Users
• SC-Captioner: Improving Image Captioning with Self-Correction by Reinforcement Learning
• Do Recommender Systems Really Leverage Multimodal Content? A Comprehensive Analysis on Multimodal Representations for Recommendation
• Gather and Trace: Rethinking Video TextVQA from an Instance-oriented Perspective
• T2UE: Generating Unlearnable Examples from Text Descriptions
• SA-GCS: Semantic-Aware Gaussian Curriculum Scheduling for UAV Vision-Language Navigation
• Phi-Ground Tech Report: Advancing Perception in GUI Grounding

Discussions

• GIT: A Generative Image-to-Text Transformer for Vision and Language [hn, 15 points, 1 comments]
• Git: A Generative Image-to-Text Transformer for Vision and Language [hn, 2 points, 0 comments]

Related

• GiT: Towards Generalist Vision Transformer through Universal Language Interface
• VL-GPT: A Generative Pre-trained Transformer for Vision and Language Understanding and Generation
• Vision Transformer: Vit and its Derivatives
• UNIT: Unifying Image and Text Recognition in One Vision Encoder
• From Show to Tell: A Survey on Deep Learning-based Image Captioning