Does equivariance matter at scale?

2024/10/30 by Johann Brehmer, Brehmer, Johann, Sönke Behrends +5 · 5 voices · 13 citations
Materials Science · Computer Science · Physics and Astronomy · #Machine Learning in Materials Science #Advanced Graph Neural Networks #Model Reduction and Neural Networks

paper · pdf · doi:10.48550/arxiv.2410.23179

Abstract

Given large datasets and sufficient compute, is it beneficial to design neural architectures for the structure and symmetries of each problem? Or is it more efficient to learn them from data? We study empirically how equivariant and non-equivariant networks scale with compute and training samples. Focusing on a benchmark problem of rigid-body interactions and on general-purpose transformer architectures, we perform a series of experiments, varying the model size, training steps, and dataset size. We find evidence for three conclusions. First, equivariance improves data efficiency, but training non-equivariant models with data augmentation can close this gap given sufficient epochs. Second, scaling with compute follows a power law, with equivariant models outperforming non-equivariant ones at each tested compute budget. Finally, the optimal allocation of a compute budget onto model size and training duration differs between equivariant and non-equivariant models.

Cited by

• Quick ViTs: Speeding up Vision Transformers through Equivariance
• Soft Geometric Inductive Bias for Object Centric Dynamics
• Coupled Cluster con MōLe: Molecular Orbital Learning for Neural Wavefunctions
• Training Dynamics of Learning 3D-Rotational Equivariance
• When is a System Discoverable from Data? Discovery Requires Chaos
• A Circular Argument : Does RoPE need to be Equivariant for Vision?
• Rotational equivariance and locality in data-driven subgrid-scale closures
• On Uncertainty Calibration for Equivariant Functions
• Conditional Clifford-Steerable CNNs with Complete Kernel Basis for PDE Modeling
• Scaling Laws and Symmetry, Evidence from Neural Force Fields
• Equivariance by Local Canonicalization: A Matter of Representation
• Identifiable Equivariant Networks are Layerwise Equivariant
• LeMat-Traj: A Scalable and Unified Dataset of Materials Trajectories for Atomistic Modeling

Discussions

• Excellent talk by @johannbrehmer.bsky.social On “Does equivariance matter at scale?” At NeurReps workshop arxiv.org/abs/2410.23179 www.neurreps.org [bsky, 63 points, 1 comments]
• The snow is gently falling outside the window, the models are training, what could be better? Two articles cool to read: Does Equivariance matter at scale? (@johannbrehmer.bsky.social et al.) arxiv. [bsky, 42 points, 0 comments]
• Tomorrow, in the reading group, Johannes Brehmer will present his paper "Does equivariance matter at scale?" arxiv.org/abs/2410.23179 Join us on zoom at 12pm ET / 6pm CET: portal.valencelabs.com/logg [bsky, 28 points, 0 comments]
• Does physical modeling matter or should we just do data-driven stuff? I feel like this paper characterizes things perfectly: arxiv.org/abs/2410.23179. Physics-based modeling is always helpful but the [bsky, 19 points, 1 comments]
• You might not be surprised to hear that equivariance improves data efficiency. But did you expect equivariant models to also be more *compute*-efficient? Learning symmetries from data costs FLOPs! [bsky, 6 points, 1 comments]

Related

• Scaling Laws and Symmetry, Evidence from Neural Force Fields
• Using and Abusing Equivariance
• Learning (Approximately) Equivariant Networks via Constrained Optimization
• The Surprising Effectiveness of Equivariant Models in Domains with Latent Symmetry
• Approximately Equivariant Neural Processes