2022/10/09 by Shraman Pramanick, Jing Li, Pramanick, Shraman +11 · 11 citations
Computer Science · Engineering · #Advanced Neural Network Applications #Artificial intelligence #Bounding overwatch #Computer Vision and Pattern Recognition (cs.CV) #Computer science #Computer vision #Domain Adaptation and Few-Shot Learning #Engineering #FOS: Computer and information sciences #Image (mathematics) #Machine Learning (cs.LG) #Minimum bounding box #Multimedia (cs.MM) #Multimodal Machine Learning Applications #Pattern recognition (psychology) #Segmentation #Transformer
paper · pdf · doi:10.48550/arxiv.2210.04135
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2022/10/09 · openalex created_date 2022/10/12 · openalex updated_date 2026/07/28
Vision-language pre-training (VLP) has recently proven highly effective for various uni- and multi-modal downstream applications. However, most existing end-to-end VLP methods use high-resolution image-text box data to perform well on fine-grained region-level tasks, such as object detection, segmentation, and referring expression comprehension. Unfortunately, such high-resolution images with accurate bounding box annotations are expensive to collect and use for supervision at scale. In this work, we propose VoLTA (Vision-Language Transformer with weakly-supervised local-feature Alignment), a new VLP paradigm that only utilizes image-caption data but achieves fine-grained region-level image understanding, eliminating the use of expensive box annotations. VoLTA adopts graph optimal transport-based weakly-supervised alignment on local image patches and text tokens to germinate an explicit, self-normalized, and interpretable low-level matching criterion. In addition, VoLTA pushes multi-modal fusion deep into the uni-modal backbones during pre-training and removes fusion-specific transformer layers, further reducing memory requirements. Extensive experiments on a wide range of vision- and vision-language downstream tasks demonstrate the effectiveness of VoLTA on fine-grained applications without compromising the coarse-grained downstream performance, often outperforming methods using significantly more caption and box annotations.