SPECTRE: An FFT-Based Efficient Drop-In Replacement to Self-Attention for Long Contexts

2025/02/25 by Jacob Fein-Ashley, Fein-Ashley, Jacob, Neelesh Gupta +6 · 22 voices
Computer Science · #Advanced Neural Network Applications #Multimodal Machine Learning Applications #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2502.18394

Abstract

Long-context transformers face significant efficiency challenges due to the quadratic cost of self-attention. However, many modern applications-from multi-turn dialogue to high-resolution vision-require contexts spanning tens of thousands of tokens. We introduce SPECTRE, a method that replaces each attention head with a fast real FFT, a content-adaptive spectral gate, and an inverse FFT, reducing per-layer complexity from O(L²) to O(Llog L) while preserving the surrounding architecture. We extend this efficiency to autoregressive generation through our Prefix-FFT cache and enhance local feature representation with an optional wavelet module that adds negligible computational overhead. Our experiments demonstrate that SPECTRE operates up to 7× faster than FlashAttention-2 on 128k-token contexts while matching or exceeding baseline performance on PG-19 language modeling and ImageNet-1k classification tasks. SPECTRE achieves these improvements by adding fewer than 6% parameters to the base model, making hundred-kilotoken context processing feasible on commodity GPUs without specialized hardware.

Citations

• Mamba: Linear-Time Sequence Modeling with Selective State Spaces
• Retentive Network: A Successor to Transformer for Large Language Models
• LongNet: Scaling Transformers to 1,000,000,000 Tokens
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• FNet: Mixing Tokens with Fourier Transforms
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
• Language Models are Few-Shot Learners
• Attention Is All You Need
• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
• An algorithm for the machine calculation of complex Fourier series

Discussions

• The FFT Strikes Back: An Efficient Alternative to Self-Attention [hn, 456 points, 168 comments]
• Low-hanging fruit: If you've already done the work of laying out the sequences in a convolution-friendly way, taking their Fourier Domain representation is a multiplication. Thing is the layouts are l [bsky, 19 points, 0 comments]
• i also like this one arxiv.org/abs/2502.183... [bsky, 4 points, 0 comments]
• FFTNet: A faster, more efficient alternative to self-attention using Fast Fourier Transform 🚀 #ML #AI https://arxiv.org/abs/2502.18394 #machinelearning #neuralnetworks #fft #self-attention #performan [bsky, 1 points, 0 comments]
• The FFT Strikes Back: An Efficient Alternative to Self-Attention (arxiv.org) Main Link | Discussion [bsky, 1 points, 0 comments]
• https://bsky.app/profile/hackernews.com.web.brid.gy/post/3lj446l2ncyj2 [bsky, 0 points, 0 comments]
• The FFT Strikes Back: An Efficient Alternative to Self-Attention [bsky, 0 points, 0 comments]
• The FFT Strikes Back: An Efficient Alternative to Self-Attention #HackerNews https://arxiv.org/abs/2502.18394 [bsky, 0 points, 0 comments]
• The FFT Strikes Back: An Efficient Alternative to Self-Attention https://arxiv.org/abs/2502.18394 https://news.ycombinator.com/item?id=43182325 [bsky, 0 points, 0 comments]
• The FFT Strikes Back: An Efficient Alternative to Self-Attention https://arxiv.org/abs/2502.18394 arxiv.org [bsky, 0 points, 0 comments]
• The FFT Strikes Back: An Efficient Alternative to Self-Attention https://arxiv.org/abs/2502.18394 [bsky, 0 points, 0 comments]
• The FFT Strikes Back: An Efficient Alternative to Self-Attention https://arxiv.org/abs/2502.18394 [comments] [153 points] [telegram] [bsky, 0 points, 0 comments]
• The FFT Strikes Back: An Efficient Alternative to Self-Attention https://arxiv.org/abs/2502.18394 [bsky, 0 points, 0 comments]
• The FFT Strikes Back: An Efficient Alternative to Self-Attention https://arxiv.org/abs/2502.18394 (http://news.ycombinator.com/item?id=43182325) [bsky, 0 points, 0 comments]
• The FFT Strikes Back: An Efficient Alternative to Self-Attention https://arxiv.org/abs/2502.18394 (https://news.ycombinator.com/item?id=43182325) [bsky, 0 points, 0 comments]
• https://arxiv.org/abs/2502.18394 この論文では、Self-Attentionの効率的な代替としてFFT（高速フーリエ変換）を用いた新しい手法を提案しています。 FFTは計算コストが低く、大規模なデータセットでも高速に処理できる可能性があります。 特に、長文テキストの処理においてSelf-Attentionと同等の性能を発揮することが期待されます。 [bsky, 0 points, 0 comments]
• The FFT Strikes Back: An Efficient Alternative to Self-Attention view on hacker news [bsky, 0 points, 0 comments]
• The FFT Strikes Back: An Efficient Alternative to Self-Attention https://arxiv.org/abs/2502.18394 (http://news.ycombinator.com/item?id=43182325) [bsky, 0 points, 0 comments]
• The FFT Strikes Back: An Efficient Alternative to Self-Attention https://arxiv.org/abs/2502.18394 (https://news.ycombinator.com/item?id=43182325) [bsky, 0 points, 0 comments]
• The FFT Strikes Back: An Efficient Alternative to Self-Attention https://arxiv.org/abs/2502.18394 (https://news.ycombinator.com/item?id=43182325) [bsky, 0 points, 0 comments]
• Ditching the quadratic complexity of transformer models would be nice. I like nice things. arxiv.org/abs/2502.18394 [bsky, 0 points, 0 comments]
• "The FFT Strikes Back: An Efficient Alternative to Self-Attention" New ways to use math in technology could save time and make things faster. This might change how we build smart programs. Article | [bsky, 0 points, 0 comments]

Related

• DistrAttention: An Efficient and Flexible Self-Attention Mechanism on Modern GPUs
• Lean Attention: Hardware-Aware Scalable Attention Mechanism for the Decode-Phase of Transformers
• Theory, Analysis, and Best Practices for Sigmoid Self-Attention
• FlashMask: Efficient and Rich Mask Extension of FlashAttention
• LIPT: Latency-aware Image Processing Transformer