Improving Lipschitz-Constrained Neural Networks by Learning Activation Functions

2022/10/28 by Stanislas Ducotterd, Alexis Goujon, Ducotterd, Stanislas +9 · 9 citations
Biochemistry, Genetics and Molecular Biology · Mathematics · #Activation function #Advanced Fluorescence Microscopy Techniques #Artificial intelligence #Artificial neural network #Computer science #Contrast (vision) #FOS: Computer and information sciences #Lipschitz continuity #Machine Learning (cs.LG) #Mathematical optimization #Mathematics #Pure mathematics #Regularization (linguistics) #Spectroscopy Techniques in Biomedical and Chemical Research

paper · pdf · doi:10.48550/arxiv.2210.16222

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2022/10/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/08

Abstract

Lipschitz-constrained neural networks have several advantages over unconstrained ones and can be applied to a variety of problems, making them a topic of attention in the deep learning community. Unfortunately, it has been shown both theoretically and empirically that they perform poorly when equipped with ReLU activation functions. By contrast, neural networks with learnable 1-Lipschitz linear splines are known to be more expressive. In this paper, we show that such networks correspond to global optima of a constrained functional optimization problem that consists of the training of a neural network composed of 1-Lipschitz linear layers and 1-Lipschitz freeform activation functions with second-order total-variation regularization. Further, we propose an efficient method to train these neural networks. Our numerical experiments show that our trained networks compare favorably with existing 1-Lipschitz neural architectures.

Cited by

• Trainable Nonexpansive Denoisers for Contractive Image Reconstruction
• Topologically-Stabilized Graph Neural Networks: Empirical Robustness Across Domains
• A Relaxed Gradient Step Denoiser for Splitting Methods in Poisson Inverse Problems
• Learning Regularization Functionals for Inverse Problems: A Comparative Study
• Multivariate Fields of Experts
• Plug and Play Splitting Techniques for Poisson Image Restoration
• Generalization bounds for score-based generative models: a synthetic proof
• Bridging GANs and Bayesian Neural Networks via Partial Stochasticity
• Statistically Undetectable Backdoors in Deep Neural Networks

Related

• Approximation of Lipschitz Functions using Deep Spline Neural Networks
• Learning Lipschitz Functions by GD-trained Shallow Overparameterized ReLU Neural Networks
• Training Certifiably Robust Neural Networks with Efficient Local Lipschitz Bounds
• A Unified Algebraic Perspective on Lipschitz Neural Networks
• 1-Lipschitz Neural Networks are more expressive with N-Activations