Relaxing Equivariance Constraints with Non-stationary Continuous Filters

2022/04/14 by Tycho F. A. van der Ouderaa, David W. Romero, van der Ouderaa, Tycho F. A. +3 · 8 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · Materials Science · Mathematics · #Artificial intelligence #Artificial neural network #Computer science #Convolution (computer science) #Equivariant map #Geometry #Homogeneous space #Machine Learning in Bioinformatics #Machine Learning in Materials Science #Mathematics #Pure mathematics #Symmetry (geometry) #Topic Modeling #cs.LG

paper · pdf · doi:10.48550/arxiv.2204.07178

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2022/04/14 · arxiv created 2022/11/13 · arxiv updated 2022/11/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06

Abstract

Equivariances provide useful inductive biases in neural network modeling, with the translation equivariance of convolutional neural networks being a canonical example. Equivariances can be embedded in architectures through weight-sharing and place symmetry constraints on the functions a neural network can represent. The type of symmetry is typically fixed and has to be chosen in advance. Although some tasks are inherently equivariant, many tasks do not strictly follow such symmetries. In such cases, equivariance constraints can be overly restrictive. In this work, we propose a parameter-efficient relaxation of equivariance that can effectively interpolate between a (i) non-equivariant linear product, (ii) a strict-equivariant convolution, and (iii) a strictly-invariant mapping. The proposed parameterisation can be thought of as a building block to allow adjustable symmetry structure in neural networks. In addition, we demonstrate that the amount of equivariance can be learned from the training data using backpropagation. Gradient-based learning of equivariance achieves similar or improved performance compared to the best value found by cross-validation and outperforms baselines with partial or strict equivariance on CIFAR-10 and CIFAR-100 image classification tasks.

Cited by

• Achieving Approximate Symmetry Is Exponentially Easier than Exact Symmetry
• Partially Equivariant Reinforcement Learning in Symmetry-Breaking Environments
• Conditional Clifford-Steerable CNNs with Complete Kernel Basis for PDE Modeling
• Latent Mixture of Symmetries for Sample-Efficient Dynamic Learning
• High-Order Deep Meta-Learning with Category-Theoretic Interpretation
• A Tale of Two Symmetries: Exploring the Loss Landscape of Equivariant Models
• A Regularization-Guided Equivariant Approach for Image Restoration
• AtlasD: Automatic Local Symmetry Discovery

Related

• Symmetry Breaking and Equivariant Neural Networks
• Learning (Approximately) Equivariant Networks via Constrained Optimization
• Learning Layer-wise Equivariances Automatically using Gradients
• Equivariance with Learned Canonicalization Functions
• Improving Equivariant Model Training via Constraint Relaxation