SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks

2020/06/18 by Fuchs, Fabian B., Worrall, Daniel E., Fischer, Volker +1 · 52 citations
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.2006.10503

Abstract

We introduce the SE(3)-Transformer, a variant of the self-attention module for 3D point clouds and graphs, which is equivariant under continuous 3D roto-translations. Equivariance is important to ensure stable and predictable performance in the presence of nuisance transformations of the data input. A positive corollary of equivariance is increased weight-tying within the model. The SE(3)-Transformer leverages the benefits of self-attention to operate on large point clouds and graphs with varying number of points, while guaranteeing SE(3)-equivariance for robustness. We evaluate our model on a toy N-body particle simulation dataset, showcasing the robustness of the predictions under rotations of the input. We further achieve competitive performance on two real-world datasets, ScanObjectNN and QM9. In all cases, our model outperforms a strong, non-equivariant attention baseline and an equivariant model without attention.

Cited by

• The Convergence Frontier: Integrating Machine Learning and High Performance Quantum Computing for Next-Generation Drug Discovery
• QE-Catalytic: A Graph-Language Multimodal Base Model for Relaxed-Energy Prediction in Catalytic Adsorption
• ChemATP: A Training-Free Chemical Reasoning Framework for Large Language Models
• A Multimodal Approach to Alzheimer's Diagnosis: Geometric Insights from Cube Copying and Cognitive Assessments
• Soft Geometric Inductive Bias for Object Centric Dynamics
• Accelerating MHC-II Epitope Discovery via Multi-Scale Prediction in Antigen Presentation
• A Single Architecture for Representing Invariance Under Any Space Group
• Drawback of Enforcing Equivariance and its Compensation via the Lens of Expressive Power
• Equivariant Diffusion for Crystal Structure Prediction
• Learning Group Actions In Disentangled Latent Image Representations
• Training Dynamics of Learning 3D-Rotational Equivariance
• Beyond Atoms: Evaluating Electron Density Representation for 3D Molecular Learning
• Generative models for crystalline materials
• GLOBE: Accurate and Generalizable PDE Surrogates using Domain-Inspired Architectures and Equivariances
• Twist and Compute: The Cost of Pose in 3D Generative Diffusion
• VEDA: 3D Molecular Generation via Variance-Exploding Diffusion with Annealing
• Hierarchical Direction Perception via Atomic Dot-Product Operators for Rotation-Invariant Point Clouds Learning
• Recurrent Equivariant Constraint Modulation: Learning Per-Layer Symmetry Relaxation from Data
• Navigating protein–nucleic acid sequence-structure landscapes with deep learning
• Direct Molecular Polarizability Prediction with SO(3) Equivariant Local Frame GNNs
• Reaction Prediction via Interaction Modeling of Symmetric Difference Shingle Sets
• Learning with Category-Equivariant Architectures for Human Activity Recognition
• Transformer Atomic Cluster Expansion: TRACE
• Knowledge Distillation of a Protein Language Model Yields a Foundational Implicit Solvent Model
• Perception Learning: A Formal Separation of Sensory Representation Learning from Decision Learning
• Equivariant Neural Networks for General Linear Symmetries on Lie Algebras
• DeepChem Equivariant: SE(3)-Equivariant Support in an Open-Source Molecular Machine Learning Library
• On Universality of Deep Equivariant Networks
• Conditional Clifford-Steerable CNNs with Complete Kernel Basis for PDE Modeling
• Controllable Graph Generation with Diffusion Models via Inference-Time Tree Search Guidance
• Transferable Generative Models Bridge Femtosecond to Nanosecond Time-Step Molecular Dynamics
• When Does Global Attention Help? A Unified Empirical Study on Atomistic Graph Learning
• ATOM: A Pretrained Neural Operator for Multitask Molecular Dynamics
• Physics-Inspired All-Pair Interaction Learning for 3D Dynamics Modeling
• Learning Generalizable Shape Completion with SIM(3) Equivariance
• Towards Understanding the Shape of Representations in Protein Language Models
• Single-Core Superscalar Optimization of Clifford Neural Layers
• Towards Foundation Models for Cryo-ET Subtomogram Analysis
• Adaptive Canonicalization with Application to Invariant Anisotropic Geometric Networks
• Clebsch-Gordan Transformer: Fast and Global Equivariant Attention
• Automatic Discovery of One-Parameter Subgroups of Lie Groups: Compact and Non-Compact Cases of SO(n) and SL(n)
• Learning Inter-Atomic Potentials without Explicit Equivariance
• Artificial Intelligence for Direct Prediction of Molecular Dynamics across Chemical Space
• Toward Embodiment Equivariant Vision-Language-Action Policy
• Beyond Ensembles: Simulating All-Atom Protein Dynamics in a Learned Latent Space
• FIRE-GNN: Force-informed, Relaxed Equivariance Graph Neural Network for Rapid and Accurate Prediction of Surface Properties
• Surfel-based 3D Registration with Equivariant SE(3) Features
• Equi-mRNA: Protein Translation Equivariant Encoding for mRNA Language Models
• BLIPs: Bayesian Learned Interatomic Potentials
• Chi-Geometry: A Library for Benchmarking Chirality Prediction of GNNs
• HemePLM-Diffuse: A Scalable Generative Framework for Protein-Ligand Dynamics in Large Biomolecular System
• Deeper Inside Deep ViT

Related

• Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs
• Iterative SE(3)-Transformers
• EquAct: An SE(3)-Equivariant Multi-Task Transformer for Open-Loop Robotic Manipulation
• SE(3)-Equivariant Attention Networks for Shape Reconstruction in Function Space
• SE3ET: SE(3)-Equivariant Transformer for Low-Overlap Point Cloud Registration