Input Hessian Regularization of Neural Networks

2020/09/14 by Waleed Mustafa, Robert A. Vandermeulen, Mustafa, Waleed +3
Computer Science · Engineering · Physics and Astronomy · #Advanced Memory and Neural Computing #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Model Reduction and Neural Networks

paper · pdf · doi:10.48550/arxiv.2009.06571

openalex publication_date 2020/09/14 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Regularizing the input gradient has shown to be effective in promoting the robustness of neural networks. The regularization of the input's Hessian is therefore a natural next step. A key challenge here is the computational complexity. Computing the Hessian of inputs is computationally infeasible. In this paper we propose an efficient algorithm to train deep neural networks with Hessian operator-norm regularization. We analyze the approach theoretically and prove that the Hessian operator norm relates to the ability of a neural network to withstand an adversarial attack. We give a preliminary experimental evaluation on the MNIST and FMNIST datasets, which demonstrates that the new regularizer can, indeed, be feasible and, furthermore, that it increases the robustness of neural networks over input gradient regularization.

Citations

• Adversarial Machine Learning at Scale
• On the (Statistical) Detection of Adversarial Examples
• Towards Deep Learning Models Resistant to Adversarial Attacks
• A Kernel Perspective for Regularizing Deep Neural Networks
• Theoretically Principled Trade-off between Robustness and Accuracy
• On Evaluating Adversarial Robustness
• Decision-Based Adversarial Attacks: Reliable Attacks Against Black-Box Machine Learning Models
• Improving the Adversarial Robustness and Interpretability of Deep Neural\n Networks by Regularizing their Input Gradients

Related

• Lipschitz regularized Deep Neural Networks generalize and are adversarially robust
• Regularizing Deep Neural Networks with Stochastic Estimators of Hessian Trace
• On Connections between Regularizations for Improving DNN Robustness
• Generalizing and Improving Jacobian and Hessian Regularization
• A Kernel Perspective for Regularizing Deep Neural Networks