2021/08/31 by Santiago Ontañón, Joshua Ainslie, Vaclav Cvicek +1 · 1 citation
Computer Science · #cs.AI #cs.CL
published as ACL 2022 · Source code: https://github.com/google-research/google-research/tree/master/compositional_transformers
arxiv created 2022/03/03 · arxiv updated 2022/03/04
Several studies have reported the inability of Transformer models to generalize compositionally, a key type of generalization in many NLP tasks such as semantic parsing. In this paper we explore the design space of Transformer models showing that the inductive biases given to the model by several design decisions significantly impact compositional generalization. Through this exploration, we identified Transformer configurations that generalize compositionally significantly better than previously reported in the literature in a diverse set of compositional tasks, and that achieve state-of-the-art results in a semantic parsing compositional generalization benchmark (COGS), and a string edit operation composition benchmark (PCFG).