2023/09/04 by Nicolas Zucchet, Zucchet, Nicolas, Seijin Kobayashi +11 · 1 voice
Computer Science · Engineering · #Advanced Memory and Neural Computing #Neural Networks and Applications #Neural Networks and Reservoir Computing #cs.LG #cs.NE
paper · pdf · doi:10.48550/arxiv.2309.01775
openalex publication_date 2023/09/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Recent architectural developments have enabled recurrent neural networks (RNNs) to reach and even surpass the performance of Transformers on certain sequence modeling tasks. These modern RNNs feature a prominent design pattern: linear recurrent layers interconnected by feedforward paths with multiplicative gating. Here, we show how RNNs equipped with these two design elements can exactly implement (linear) self-attention, the main building block of Transformers. By reverse-engineering a set of trained RNNs, we find that gradient descent in practice discovers our construction. In particular, we examine RNNs trained to solve simple in-context learning tasks on which Transformers are known to excel and find that gradient descent instills in our RNNs the same attention-based in-context learning algorithm used by Transformers. Our findings highlight the importance of multiplicative interactions in neural networks and suggest that certain RNNs might be unexpectedly implementing attention under the hood.