Rethinking Attention: Polynomial Alternatives to Softmax in Transformers
2024/10/24 by Hemanth Saratchandran, Jianqiao Zheng, Saratchandran, Hemanth +7 · 6 voices · 1 citation
#cs.LG #cs.CV #stat.ML
paper · pdf · doi:10.48550/arxiv.2410.18613
Abstract
This paper questions whether the strong performance of softmax attention in transformers stems from producing a probability distribution over inputs. Instead, we argue that softmax's effectiveness lies in its implicit regularization of the Frobenius norm of the attention matrix, which stabilizes training. Motivated by this, we explore alternative activations, specifically polynomials, that achieve a similar regularization effect. Our theoretical analysis shows that certain polynomials can serve as effective substitutes for softmax, achieving strong performance across transformer applications despite violating softmax's typical properties of positivity, normalization, and sparsity. Extensive experiments support these findings, offering a new perspective on attention mechanisms.
Cited by
Discussions
- *Rethinking Softmax: Self-Attention with Polynomial Activations*
They show that using softmax in the attention computation upper-bounds the Frobenius norm of the attention matrix, and similar results [bsky, 7 points, 0 comments]
- See e.g. arxiv.org/abs/2410.18613 that recently showed that we can replace softmax attention with alternatives that do not satisfy the properties we intuitively assign to it, and yet these models seem [bsky, 6 points, 1 comments]
- Rethinking Softmax: Self-Attention with Polynomial Activations [hn, 2 points, 0 comments]
- @davidpicard.eurosky.social ? I remember a paper that explored a variety of normalizations for attention (l1, ...) but couldn't retrieve it. A recent version that somehow my brain connects with David [bsky, 1 points, 1 comments]
- Softmax in the final layer of an ANN might produce a probability distribution but inside an attenion head, it seems to be performing a normalising function. I think this cool new paper supports this c [bsky, 1 points, 1 comments]
- Transformer “attention probabilities” might be theater. The real win was hidden regularization of the attention matrix. If a dumb polynomial can replace softmax, half the lore around attention was car [bsky, 0 points, 0 comments]
Related