Meta-Learning Symmetries by Reparameterization

2020/07/06 by Allan Zhou, Zhou, Allan, Tom Knowles +3 · 10 citations
Computer Science · Mathematics · #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Machine Learning and Data Classification #Multimodal Machine Learning Applications #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.2007.02933

ICLR 2021

openalex publication_date 2020/07/06 · arxiv created 2021/03/30 · arxiv updated 2021/03/31 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Many successful deep learning architectures are equivariant to certain transformations in order to conserve parameters and improve generalization: most famously, convolution layers are equivariant to shifts of the input. This approach only works when practitioners know the symmetries of the task and can manually construct an architecture with the corresponding equivariances. Our goal is an approach for learning equivariances from data, without needing to design custom task-specific architectures. We present a method for learning and encoding equivariances into networks by learning corresponding parameter sharing patterns from data. Our method can provably represent equivariance-inducing parameter sharing for any finite group of symmetry transformations. Our experiments suggest that it can automatically learn to encode equivariances to common transformations used in image processing tasks. We provide our experiment code at https://github.com/AllanYangZhou/metalearning-symmetries.

Citations

• Adam: A Method for Stochastic Optimization
• Tensor Decompositions and Applications
• Gradient-based learning applied to document recognition
• ImageNet Large Scale Visual Recognition Challenge
• Human-level concept learning through probabilistic program induction
• Neural Architecture Search with Reinforcement Learning
• Equivariance Through Parameter-Sharing
• Prototypical Networks for Few-shot Learning
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• Meta-SGD: Learning to Learn Quickly for Few-Shot Learning
• SMASH: One-Shot Model Architecture Search through HyperNetworks
• On the Generalization of Equivariance and Convolution in Neural Networks to the Action of Compact Groups
• AutoAugment: Learning Augmentation Policies from Data
• Meta-Learning with Latent Embedding Optimization
• How to train your MAML
• An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution
• General E(2)-Equivariant Steerable CNNs
• Torchmeta: A Meta-Learning library for PyTorch
• Generalized Inner Loop Meta-Learning
• Optimizing Millions of Hyperparameters by Implicit Differentiation
• Lagrangian Neural Networks
• Learning Invariances in Neural Networks
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

Cited by

• Discovering Symmetry Groups with Flow Matching
• Reinforcement Learning Using known Invariances
• Explicit Discovery of Nonlinear Symmetries from Dynamic Data
• Latent Mixture of Symmetries for Sample-Efficient Dynamic Learning
• Learning Equivariant Functions via Quadratic Forms
• Geometrically Constrained and Token-Based Probabilistic Spatial Transformers
• EquiReg: Equivariance Regularized Diffusion for Inverse Problems
• RECON: Robust symmetry discovery via Explicit Canonical Orientation Normalization
• Addressing the Topological Defects of Disentanglement via Distributed\n Operators
• AtlasD: Automatic Local Symmetry Discovery

Related

• Symmetry From Scratch: Group Equivariance as a Supervised Learning Task
• Learning Probabilistic Symmetrization for Architecture Agnostic Equivariance
• Learning Layer-wise Equivariances Automatically using Gradients
• Learning Symmetric Embeddings for Equivariant World Models
• Using and Abusing Equivariance