Emergent Equivariance in Deep Ensembles

2024/03/05 by Jan E. Gerken, Gerken, Jan E., Pan Kessel +1 · 6 citations
Physics and Astronomy · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Opinion Dynamics and Social Influence

paper · pdf · doi:10.48550/arxiv.2403.03103

openalex publication_date 2024/03/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We show that deep ensembles become equivariant for all inputs and at all training times by simply using data augmentation. Crucially, equivariance holds off-manifold and for any architecture in the infinite width limit. The equivariance is emergent in the sense that predictions of individual ensemble members are not equivariant but their collective prediction is. Neural tangent kernel theory is used to derive this result and we verify our theoretical insights using detailed numerical experiments.

Cited by

• Implicit Bias and Invariance: How Hopfield Networks Efficiently Learn Graph Orbits
• Finite-Width Neural Tangent Kernels from Feynman Diagrams
• Do we need equivariant models for molecule generation?
• Not All Explanations for Deep Learning Phenomena Are Equally Valuable
• Understanding Learning Invariance in Deep Linear Networks
• How Ensembles of Distilled Policies Improve Generalisation in Reinforcement Learning

Related

• Ensembles provably learn equivariance through data augmentation
• Data Augmentation vs. Equivariant Networks: A Theory of Generalization on Dynamics Forecasting
• Identifiable Equivariant Networks are Layerwise Equivariant
• Equivariant Neural Tangent Kernels
• Topographic VAEs learn Equivariant Capsules