SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks

2020/06/18 by Fabian B. Fuchs, Daniel E. Worrall, Fuchs, Fabian B. +5 · 94 citations
Computer Science · Engineering · Materials Science · Mathematics · #3D Shape Modeling and Analysis #Advanced Neural Network Applications #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning in Materials Science #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2006.10503

openalex publication_date 2020/06/18 · arxiv created 2020/11/24 · arxiv updated 2020/11/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce the SE(3)-Transformer, a variant of the self-attention module for 3D point clouds and graphs, which is equivariant under continuous 3D roto-translations. Equivariance is important to ensure stable and predictable performance in the presence of nuisance transformations of the data input. A positive corollary of equivariance is increased weight-tying within the model. The SE(3)-Transformer leverages the benefits of self-attention to operate on large point clouds and graphs with varying number of points, while guaranteeing SE(3)-equivariance for robustness. We evaluate our model on a toy N-body particle simulation dataset, showcasing the robustness of the predictions under rotations of the input. We further achieve competitive performance on two real-world datasets, ScanObjectNN and QM9. In all cases, our model outperforms a strong, non-equivariant attention baseline and an equivariant model without attention.

Citations

• Engineering Applications of Noncommutative Harmonic Analysis: With Emphasis on Rotation and Motion Groups
• Neural Message Passing for Quantum Chemistry
• Tensor field networks: Rotation- and translation-equivariant neural networks for 3D point clouds
• On the Limitations of Representing Functions on Sets
• Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networks
• PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space
• General E(2)-Equivariant Steerable CNNs
• Dynamic Graph CNN for Learning on Point Clouds

Cited by

• The Convergence Frontier: Integrating Machine Learning and High Performance Quantum Computing for Next-Generation Drug Discovery
• QE-Catalytic: A Graph-Language Multimodal Base Model for Relaxed-Energy Prediction in Catalytic Adsorption
• ChemATP: A Training-Free Chemical Reasoning Framework for Large Language Models
• A Multimodal Approach to Alzheimer's Diagnosis: Geometric Insights from Cube Copying and Cognitive Assessments
• Soft Geometric Inductive Bias for Object Centric Dynamics
• Accelerating MHC-II Epitope Discovery via Multi-Scale Prediction in Antigen Presentation
• A Single Architecture for Representing Invariance Under Any Space Group
• Drawback of Enforcing Equivariance and its Compensation via the Lens of Expressive Power
• Equivariant Diffusion for Crystal Structure Prediction
• Learning Group Actions In Disentangled Latent Image Representations
• Training Dynamics of Learning 3D-Rotational Equivariance
• Beyond Atoms: Evaluating Electron Density Representation for 3D Molecular Learning
• Generative models for crystalline materials
• GLOBE: Accurate and Generalizable PDE Surrogates using Domain-Inspired Architectures and Equivariances
• Twist and Compute: The Cost of Pose in 3D Generative Diffusion
• VEDA: 3D Molecular Generation via Variance-Exploding Diffusion with Annealing
• Hierarchical Direction Perception via Atomic Dot-Product Operators for Rotation-Invariant Point Clouds Learning
• Recurrent Equivariant Constraint Modulation: Learning Per-Layer Symmetry Relaxation from Data
• Navigating protein–nucleic acid sequence-structure landscapes with deep learning
• Direct Molecular Polarizability Prediction with SO(3) Equivariant Local Frame GNNs
• Reaction Prediction via Interaction Modeling of Symmetric Difference Shingle Sets
• Learning with Category-Equivariant Architectures for Human Activity Recognition
• Transformer Atomic Cluster Expansion: TRACE
• Knowledge Distillation of a Protein Language Model Yields a Foundational Implicit Solvent Model
• Perception Learning: A Formal Separation of Sensory Representation Learning from Decision Learning
• Equivariant Neural Networks for General Linear Symmetries on Lie Algebras
• DeepChem Equivariant: SE(3)-Equivariant Support in an Open-Source Molecular Machine Learning Library
• On Universality of Deep Equivariant Networks
• Conditional Clifford-Steerable CNNs with Complete Kernel Basis for PDE Modeling
• Controllable Graph Generation with Diffusion Models via Inference-Time Tree Search Guidance
• Transferable Generative Models Bridge Femtosecond to Nanosecond Time-Step Molecular Dynamics
• When Does Global Attention Help? A Unified Empirical Study on Atomistic Graph Learning
• ATOM: A Pretrained Neural Operator for Multitask Molecular Dynamics
• PAINET: A Principled Efficient Transformer for 3D Dynamics Modeling
• Learning Generalizable Shape Completion with SIM(3) Equivariance
• Towards Understanding the Shape of Representations in Protein Language Models
• Single-Core Superscalar Optimization of Clifford Neural Layers
• Towards Foundation Models for Cryo-ET Subtomogram Analysis
• Adaptive Canonicalization with Application to Invariant Anisotropic Geometric Networks
• Clebsch-Gordan Transformer: Fast and Global Equivariant Attention
• Interpretable Discovery of One-parameter Subgroups: A Modular Framework for Elliptical, Hyperbolic, and Parabolic Symmetries
• Learning Inter-Atomic Potentials without Explicit Equivariance
• Artificial Intelligence for Direct Prediction of Molecular Dynamics across Chemical Space
• Toward Embodiment Equivariant Vision-Language-Action Policy
• Beyond Ensembles: Simulating All-Atom Protein Dynamics in a Learned Latent Space
• FIRE-GNN: Force-informed, Relaxed Equivariance Graph Neural Network for Rapid and Accurate Prediction of Surface Properties
• Surfel-based 3D Registration with Equivariant SE(3) Features
• Equi-mRNA: Protein Translation Equivariant Encoding for mRNA Language Models
• BLIPs: Bayesian Learned Interatomic Potentials
• Chi-Geometry: A Library for Benchmarking Chirality Prediction of GNNs
• HemePLM-Diffuse: A Scalable Generative Framework for Protein-Ligand Dynamics in Large Biomolecular System
• Deeper Inside Deep ViT
• Efficient Prediction of SO(3)-Equivariant Hamiltonian Matrices via SO(2) Local Frames
• Equivariant Volumetric Grasping
• DualEquiNet: A Dual-Space Hierarchical Equivariant Network for Large Biomolecules
• GGMotion: Group Graph Dynamics-Kinematics Networks for Human Motion Prediction
• Quaternionic Response Geometry for Proteins: Toward a Noncommutative Theory of Ordered Deformations
• Learning Manifolds in High-D Point Embedding for Anisotropic Surface Approximation from Unstructured Point Clouds
• Tensor Decomposition Networks for Fast Machine Learning Interatomic Potential Computations
• Rotational Sampling: A Plug-and-Play Encoder for Rotation-Invariant 3D Molecular GNNs
• Unraveling the Potential of Diffusion Models in Small Molecule Generation
• Fast and Distributed Equivariant Graph Neural Networks by Virtual Node Learning
• Generative Modeling of Full-Atom Protein Conformations using Latent Diffusion on Graph Embeddings
• Deep Equivariant Multi-Agent Control Barrier Functions
• GeoRecon: Graph-Level Representation Learning for 3D Molecules via Reconstruction-Based Pretraining
• GLGENN: A Novel Parameter-Light Equivariant Neural Networks Architecture Based on Clifford Geometric Algebras
• Learning dissection trajectories from expert surgical videos via imitation learning with equivariant diffusion
• FA-KPConv: Introducing Euclidean Symmetries to KPConv via Frame Averaging
• ModuLM: Enabling Modular and Multimodal Molecular Relational Learning with Large Language Models
• General Protein Pretraining or Domain-Specific Designs? Benchmarking Protein Modeling on Realistic Applications
• Equivariant Spherical Transformer for Efficient Molecular Modeling
• EquiReg: Equivariance Regularized Diffusion for Inverse Problems
• Geometric Hyena Networks for Large-scale Equivariant Learning
• EquAct: An SE(3)-Equivariant Multi-Task Transformer for Open-Loop Robotic Manipulation
• BindEnergyCraft: Casting Protein Structure Predictors as Energy-Based Models for Binder Design
• xChemAgents: Agentic AI for Explainable Quantum Chemistry
• Scalable Generation of Spatial Transcriptomics from Histology Images via Whole-Slide Flow Matching
• Equivariant Flow Matching for Point Cloud Assembly
• STFlow: Data-Coupled Flow Matching for Geometric Trajectory Simulation
• LieStoNet: Learning Lie Symmetries from Spatiotemporal Data for Stochastic Dynamical Systems
• Trajectory Prediction using Equivariant Continuous Convolution
• Discrete Neural Flow Samplers with Locally Equivariant Transformer
• Fourier-Invertible Neural Encoder (FINE) for Homogeneous Flows
• Learning (Approximately) Equivariant Networks via Constrained Optimization
• Understanding the Capabilities of Molecular Graph Neural Networks in Materials Science Through Multimodal Learning and Physical Context Encoding
• Attention on the Sphere
• EDBench: Large-Scale Electron Density Data for Molecular Modeling
• Quotient Complex Transformer (QCformer) for Perovskite Data Analysis
• RISE: Radius of Influence based Subgraph Extraction for 3D Molecular Graph Explanation
• Learning Hamiltonian Flow Maps: Mean Flow Consistency for Large-Timestep Molecular Dynamics
• Cross-Chirality Generalization by Axial Vectors for Hetero-Chiral Protein-Peptide Interaction Design
• Foundations of Equivariant Deep Learning: Unifying Graph and Sheaf Neural Networks
• Accurate and generalizable protein-ligand binding affinity prediction with geometric deep learning
• HFBRI-MAE: Handcrafted Feature Based Rotation-Invariant Masked Autoencoder for 3D Point Cloud Analysis

Related

• Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs
• Iterative SE(3)-Transformers
• EquAct: An SE(3)-Equivariant Multi-Task Transformer for Open-Loop Robotic Manipulation
• SE(3)-Equivariant Attention Networks for Shape Reconstruction in Function Space
• SE3ET: SE(3)-Equivariant Transformer for Low-Overlap Point Cloud Registration