Perspectives and Prospects on Transformer Architecture for Cross-Modal Tasks with Language and Vision

2021/03/06 by Andrew Shin, Masato Ishii, Shin, Andrew +3 · 1 citation
Arts and Humanities · Computer Science · #Advanced Image and Video Retrieval Techniques #Multimodal Machine Learning Applications #Subtitles and Audiovisual Media #cs.CL #cs.CV

paper · pdf · doi:10.48550/arxiv.2103.04037

Accepted for publication by International Journal of Computer Vision (IJCV)

arxiv created 2021/11/09 · arxiv updated 2021/11/10

Abstract

Transformer architectures have brought about fundamental changes to computational linguistic field, which had been dominated by recurrent neural networks for many years. Its success also implies drastic changes in cross-modal tasks with language and vision, and many researchers have already tackled the issue. In this paper, we review some of the most critical milestones in the field, as well as overall trends on how transformer architecture has been incorporated into visuolinguistic cross-modal tasks. Furthermore, we discuss its current limitations and speculate upon some of the prospects that we find imminent.

Cited by

• DIMT25@ICDAR2025: HW-TSC's End-to-End Document Image Machine Translation System Leveraging Large Vision-Language Model

Related

• Vision Language Transformers: A Survey
• Vision-and-Language or Vision-for-Language? On Cross-Modal Influence in Multimodal Transformers
• Speech Recognition Transformers: Topological-lingualism Perspective
• A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks
• Survey: Transformer-based Models in Data Modality Conversion