Syntax-Guided Transformers: Elevating Compositional Generalization and Grounding in Multimodal Environments

2023/11/07 by Danial Kamali, Kamali, Danial, Parisa Kordjamshidi +1
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Robotics (cs.RO) #Speech and dialogue systems #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2311.04364

openalex publication_date 2023/11/07 · openalex created_date 2023/11/10 · openalex updated_date 2026/07/28

Abstract

Compositional generalization, the ability of intelligent models to extrapolate understanding of components to novel compositions, is a fundamental yet challenging facet in AI research, especially within multimodal environments. In this work, we address this challenge by exploiting the syntactic structure of language to boost compositional generalization. This paper elevates the importance of syntactic grounding, particularly through attention masking techniques derived from text input parsing. We introduce and evaluate the merits of using syntactic information in the multimodal grounding problem. Our results on grounded compositional generalization underscore the positive impact of dependency parsing across diverse tasks when utilized with Weight Sharing across the Transformer encoder. The results push the state-of-the-art in multimodal grounding and parameter-efficient modeling and provide insights for future research.

Related

• When Can Transformers Ground and Compose: Insights from Compositional Generalization Benchmarks
• Towards Equipping Transformer with the Ability of Systematic Compositionality
• Analyzing the Inner Workings of Transformers in Compositional Generalization
• The Impact of Depth on Compositional Generalization in Transformer Language Models
• A Systematic Study of Compositional Syntactic Transformer Language Models