Self-Attention Between Datapoints: Going Beyond Individual Input-Output Pairs in Deep Learning

2021/06/04 by Jannik Kossen, Neil Band, Kossen, Jannik +9 · 10 citations
Computer Science · #Adversarial Robustness in Machine Learning #Anomaly Detection Techniques and Applications #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · pdf · doi:10.48550/arxiv.2106.02584

openalex publication_date 2021/06/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We challenge a common assumption underlying most supervised deep learning: that a model makes a prediction depending only on its parameters and the features of a single input. To this end, we introduce a general-purpose deep learning architecture that takes as input the entire dataset instead of processing one datapoint at a time. Our approach uses self-attention to reason about relationships between datapoints explicitly, which can be seen as realizing non-parametric models using parametric attention mechanisms. However, unlike conventional non-parametric models, we let the model learn end-to-end from the data how to make use of other datapoints for prediction. Empirically, our models solve cross-datapoint lookup and complex reasoning tasks unsolvable by traditional deep learning models. We show highly competitive results on tabular data, early results on CIFAR-10, and give insight into how the model makes use of the interactions between points.

Citations

• An Introduction to Kernel and Nearest-Neighbor Nonparametric Regression
• A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting
• Gradient-based learning applied to document recognition
• mice: Multivariate Imputation by Chained Equations inR
• Multidimensional binary search trees used for associative searching
• Human-level concept learning through probabilistic program induction
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Evaluating Uncertainty Quantification in End-to-End Autonomous Driving Control
• Generating Long Sequences with Sparse Transformers
• Deep Gaussian Processes for Regression using Approximate Expectation Propagation
• Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks
• Detecting Functionality-Specific Vulnerabilities via Retrieving Individual Functionality-Equivalent APIs in Open-Source Repositories
• Lookahead Optimizer: k steps forward, 1 step back
• PyTorch: An Imperative Style, High-Performance Deep Learning Library
• Axial Attention in Multidimensional Transformers
• Longformer: The Long-Document Transformer
• Language Models are Few-Shot Learners
• Array programming with NumPy
• LieTransformer: Equivariant self-attention for Lie Groups
• Learning Multiple Layers of Features from Tiny Images
• Perceiver: General Perception with Iterative Attention

Cited by

• Quantitative Verification of Fairness in Tree Ensembles
• Transfer Learning with Distance Covariance for Random Forest: Error Bounds and an EHR Application
• The Role of Feature Interactions in Graph-based Tabular Deep Learning
• LAKEGEN: A LLM-based Tabular Corpus Generator for Evaluating Dataset Discovery in Data Lakes
• Estimating Interventional Distributions with Uncertain Causal Graphs through Meta-Learning
• Zero-Shot Contextual Embeddings via Offline Synthetic Corpus Generation
• Investigating Mask-aware Prototype Learning for Tabular Anomaly Detection
• Foundation Models for Tabular Data within Systemic Contexts Need Grounding
• TabSTAR: A Tabular Foundation Model for Tabular Data with Text Fields
• LLM Meeting Decision Trees on Tabular Data

Related

• The Origin of Self-Attention: Pairwise Affinity Matrices in Feature Selection and the Emergence of Self-Attention
• Mine Your Own vieW: Self-Supervised Learning Through Across-Sample\n Prediction
• Deep-and-Wide Learning: Enhancing Data-Driven Inference via Synergistic Learning of Inter- and Intra-Data Representations
• Causal Attention for Unbiased Visual Recognition
• Deep Learning Through the Lens of Example Difficulty