Group Orthogonalization Regularization For Vision Models Adaptation and Robustness

2023/06/16 by Kurtz, Yoav, Bar, Noga, Giryes, Raja · 1 citation
#Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2306.10001

Abstract

As neural networks become deeper, the redundancy within their parameters increases. This phenomenon has led to several methods that attempt to reduce the correlation between convolutional filters. We propose a computationally efficient regularization technique that encourages orthonormality between groups of filters within the same layer. Our experiments show that when incorporated into recent adaptation methods for diffusion models and vision transformers (ViTs), this regularization improves performance on downstream tasks. We further show improved robustness when group orthogonality is enforced during adversarial training. Our code is available at https://github.com/YoavKurtz/GOR.

Cited by

• Non-Negative Stiefel Approximating Flow: Orthogonalish Matrix Optimization for Interpretable Embeddings

Related

• Gradient Regularization Improves Accuracy of Discriminative Models
• Robust Learning with Frequency Domain Regularization
• Learn to Preserve and Diversify: Parameter-Efficient Group with Orthogonal Regularization for Domain Generalization
• Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?
• Group and Shuffle: Efficient Structured Orthogonal Parametrization