A Transformer with Stack Attention

2024/05/07 by Jiaoda Li, Li, Jiaoda, Jennifer C. White +5 · 2 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Sensor Technology and Measurement Systems

paper · pdf · doi:10.48550/arxiv.2405.04515

openalex publication_date 2024/05/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Natural languages are believed to be (mildly) context-sensitive. Despite underpinning remarkably capable large language models, transformers are unable to model many context-free language tasks. In an attempt to address this limitation in the modeling power of transformer-based language models, we propose augmenting them with a differentiable, stack-based attention mechanism. Our stack-based attention mechanism can be incorporated into any transformer-based language model and adds a level of interpretability to the model. We show that the addition of our stack-based attention mechanism enables the transformer to model some, but not all, deterministic context-free languages.

Cited by

• Tree Matching Networks for Natural Language Inference: Parameter-Efficient Semantic Understanding via Dependency Parse Trees
• StackTrans: From Large Language Model to Large Pushdown Automata Model

Related

• Stack Attention: Improving the Ability of Transformers to Model Hierarchical Patterns
• Evaluating Transformer's Ability to Learn Mildly Context-Sensitive Languages
• StackTrans: From Large Language Model to Large Pushdown Automata Model
• Attention-likelihood relationship in transformers
• Skip-Layer Attention: Bridging Abstract and Detailed Dependencies in Transformers