Reservoir Transformers

2020/12/30 by Sheng Shen, Shen, Sheng, Alexei Baevski +9 · 4 citations
Computer Science · #Neural Networks and Reservoir Computing #Neural Networks and Applications #Machine Learning and ELM

paper · pdf · doi:10.48550/arxiv.2012.15045

Abstract

We demonstrate that transformers obtain impressive performance even when some of the layers are randomly initialized and never updated. Inspired by old and well-established ideas in machine learning, we explore a variety of non-linear "reservoir" layers interspersed with regular transformer layers, and show improvements in wall-clock compute time until convergence, as well as overall performance, on various machine translation and (masked) language modelling tasks.

Citations

• Gradient-based learning applied to document recognition
• Recent advances in physical reservoir computing: A review
• Random Feature Attention

Cited by

• Reservoir Computing inspired Matrix Multiplication-free Language Model
• Echo Flow Networks
• ResFormer: All-Time Reservoir Memory for Long Sequence Classification
• Frozen in Time: Parameter-Efficient Time Series Transformers via Reservoir-Induced Feature Expansion and Fixed Random Dynamics

Related

• Transformer Layers as Painters
• Transformer tricks: Precomputing the first layer
• Beyond Scaling Laws: Understanding Transformer Performance with Associative Memory
• Improving Transformer Models by Reordering their Sublayers
• What's Hidden in a One-layer Randomly Weighted Transformer?