An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architectures

2025/01/14 by Thibaut Boissin, Boissin, Thibaut, Franck Mamalet +10 · 2 voices · 5 citations
Computer Science · Mathematics · #Adversarial Robustness in Machine Learning #Algorithm #Anomaly Detection Techniques and Applications #Artificial Intelligence (cs.AI) #Artificial intelligence #Artificial neural network #Computer science #Convolution (computer science) #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Mathematical analysis #Mathematics #Neural and Evolutionary Computing (cs.NE) #Scheme (mathematics) #Theoretical computer science #cs.AI #cs.NE

paper · pdf · doi:10.48550/arxiv.2501.07930

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/01/14 · arxiv published 2025/01/14 · arxiv updated 2025/06/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/01

Abstract

Orthogonal convolutional layers are valuable components in multiple areas of machine learning, such as adversarial robustness, normalizing flows, GANs, and Lipschitz-constrained models. Their ability to preserve norms and ensure stable gradient propagation makes them valuable for a large range of problems. Despite their promise, the deployment of orthogonal convolution in large-scale applications is a significant challenge due to computational overhead and limited support for modern features like strides, dilations, group convolutions, and transposed convolutions. In this paper, we introduce AOC (Adaptative Orthogonal Convolution), a scalable method that extends a previous method (BCOP), effectively overcoming existing limitations in the construction of orthogonal convolutions. This advancement unlocks the construction of architectures that were previously considered impractical. We demonstrate through our experiments that our method produces expressive models that become increasingly efficient as they scale. To foster further advancement, we provide an open-source python package implementing this method, called Orthogonium ( https://github.com/deel-ai/orthogonium ) .

Cited by

• Fast and Flexible Robustness Certificates for Semantic Segmentation
• CAGE: Certified Authorization under Typed-Return Uncertainty for Tool-Using Agents
• Efficient Robust Conformal Prediction via Lipschitz-Bounded Networks
• Robustness as Architecture: Designing IQA Models to Withstand Adversarial Perturbations
• Distributed Retraction-Free and Communication-Efficient Optimization on the Stiefel Manifold
• HOFT: Householder Orthogonal Fine-tuning

Discussions

• 'An Adaptive Orthogonal Convolution Scheme for Efficient and Flexible CNN Architectures' Thibaut Boissin · Franck Mamalet · @thomasfel.bsky.social · Agustin Picard · Thomas Massena · Mathieu Serrurie [bsky, 2 points, 1 comments]
• Adaptive Orthogonal Convolution Scheme for Efficient, Flexible CNN Architectures [hn, 1 points, 0 comments]

Related

• Orthogonal Convolutional Neural Networks
• Scaling-up Diverse Orthogonal Convolutional Networks with a Paraunitary Framework
• Orthogonalizing Convolutional Layers with the Cayley Transform
• Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?
• Skew Orthogonal Convolutions