2021/07/16 by Krzysztof Choromański, Han Lin, Choromanski, Krzysztof +17 · 5 citations
Mathematics · Neuroscience · Physics and Astronomy · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Photoreceptor and optogenetics research #Quantum optics and atomic interactions #Random Matrices and Applications
paper · pdf · doi:10.48550/arxiv.2107.07999
openalex publication_date 2021/07/16 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
In this paper we provide, to the best of our knowledge, the first comprehensive approach for incorporating various masking mechanisms into Transformers architectures in a scalable way. We show that recent results on linear causal attention (Choromanski et al., 2021) and log-linear RPE-attention (Luo et al., 2021) are special cases of this general mechanism. However by casting the problem as a topological (graph-based) modulation of unmasked attention, we obtain several results unknown before, including efficient d-dimensional RPE-masking and graph-kernel masking. We leverage many mathematical techniques ranging from spectral analysis through dynamic programming and random walks to new algorithms for solving Markov processes on graphs. We provide a corresponding empirical evaluation.