2022/11/01 by Anuj Diwan, Layne Berry, Diwan, Anuj +7 · 7 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #cs.CL #cs.CV
paper · pdf · doi:10.48550/arxiv.2211.00768
Accepted at EMNLP 2022. We release our annotation and code at https://github.com/ajd12342/why-winoground-hard . 15 pages, 3 figures
openalex publication_date 2022/11/01 · openalex created_date 2022/11/08 · arxiv created 2022/12/03 · arxiv updated 2022/12/06 · openalex updated_date 2026/07/28
Recent visuolinguistic pre-trained models show promising progress on various end tasks such as image retrieval and video captioning. Yet, they fail miserably on the recently proposed Winoground dataset, which challenges models to match paired images and English captions, with items constructed to overlap lexically but differ in meaning (e.g., "there is a mug in some grass" vs. "there is some grass in a mug"). By annotating the dataset using new fine-grained tags, we show that solving the Winoground task requires not just compositional language understanding, but a host of other abilities like commonsense reasoning or locating small, out-of-focus objects in low-resolution images. In this paper, we identify the dataset's main challenges through a suite of experiments on related tasks (probing task, image retrieval task), data augmentation, and manual inspection of the dataset. Our analysis suggests that a main challenge in visuolinguistic models may lie in fusing visual and textual representations, rather than in compositional language understanding. We release our annotation and code at https://github.com/ajd12342/why-winoground-hard .