Enhancing the Transformer with Explicit Relational Encoding for Math Problem Solving

2019/10/15 by Schlag, Imanol, Smolensky, Paul, Fernandez, Roland +3 · 1 citation
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.1910.06611

Abstract

We incorporate Tensor-Product Representations within the Transformer in order to better support the explicit representation of relation structure. Our Tensor-Product Transformer (TP-Transformer) sets a new state of the art on the recently-introduced Mathematics Dataset containing 56 categories of free-form math word-problems. The essential component of the model is a novel attention mechanism, called TP-Attention, which explicitly encodes the relations between each Transformer cell and the other cells from which values have been retrieved by attention. TP-Attention goes beyond linear combination of retrieved values, strengthening representation-building and resolving ambiguities introduced by multiple layers of standard attention. The TP-Transformer's attention maps give better insights into how it is capable of solving the Mathematics Dataset's challenging problems. Pretrained models and code will be made available after publication.

Cited by

• Beyond Sliding Windows: Learning to Manage Memory in Non-Markovian Environments

Related

• Neural Attention: A Novel Mechanism for Enhanced Expressive Power in Transformer Models
• Theoretical Constraints on the Expressive Power of RoPE-based Tensor Attention Transformers
• Relational Attention: Generalizing Transformers for Graph-Structured Tasks
• Abstractors and relational cross-attention: An inductive bias for explicit relational reasoning in Transformers
• Expanding Expressivity in Transformer Models with MöbiusAttention