Sobolev Training for Neural Networks

2017/06/15 by Wojciech Marian Czarnecki, Simon Osindero, Czarnecki, Wojciech Marian +7 · 1 voice · 15 citations
Computer Science · #Adversarial Robustness in Machine Learning #FOS: Computer and information sciences #Generative Adversarial Networks and Image Synthesis #Machine Learning (cs.LG) #Machine Learning and Data Classification #cs.LG

paper · pdf · doi:10.48550/arxiv.1706.04859

openalex publication_date 2017/06/15 · arxiv published 2017/06/15 · arxiv updated 2017/07/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

At the heart of deep learning we aim to use neural networks as function approximators - training them to produce outputs from inputs in emulation of a ground truth function or data creation process. In many cases we only have access to input-output pairs from the ground truth, however it is becoming more common to have access to derivatives of the target output with respect to the input - for example when the ground truth function is itself a neural network such as in network compression or distillation. Generally these target derivatives are not computed, or are ignored. This paper introduces Sobolev Training for neural networks, which is a method for incorporating these target derivatives in addition the to target values while training. By optimising neural networks to not only approximate the function's outputs but also the function's derivatives we encode additional information about the target function within the parameters of the neural network. Thereby we can improve the quality of our predictors, as well as the data-efficiency and generalization capabilities of our learned function approximation. We provide theoretical justifications for such an approach as well as examples of empirical evidence on three distinct domains: regression on classical optimisation datasets, distilling policies of an agent playing Atari, and on large-scale applications of synthetic gradients. In all three domains the use of Sobolev Training, employing target derivatives in addition to target values, results in models with higher accuracy and stronger generalisation.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding
• Decoupled Neural Interfaces using Synthetic Gradients
• Exploiting gradients and Hessians in Bayesian optimization and Bayesian quadrature

Cited by

• Simultaneous Approximation of the Score Function and Its Derivatives by Deep Neural Networks
• DeepSVM: Learning Stochastic Volatility Models with Physics-Informed Deep Operator Networks
• Hankel-FNO: Fast Underwater Acoustic Charting Via Physics-Encoded Fourier Neural Operator
• First-order Sobolev Reinforcement Learning
• Precise, efficient and flexible modeling of crystallizing elastomers based on physics-augmented neural networks
• A physics-augmented neural network framework for finite strain incompressible viscoelasticity
• Distance Marching for Generative Modeling
• CODED-SMOOTHING: Coding Theory Helps Generalization
• Hybrid Layer-Wise ANN-SNN With Surrogate Spike Encoding-Decoding Structure
• Sobolev acceleration for neural networks
• Data Denoising and Derivative Estimation for Data-Driven Modeling of Nonlinear Dynamical Systems
• HyPINO: Multi-Physics Neural Operators via HyperPINNs and the Method of Manufactured Solutions
• SSBE-PINN: A Sobolev Boundary Scheme Boosting Stability and Accuracy in Elliptic/Parabolic PDE Learning
• Algebraically Observable Physics-Informed Neural Network and its Application to Epidemiological Modelling
• SlotMatch: Distilling Object-Centric Representations for Unsupervised Video Segmentation

Discussions

• Sobolev Training for Neural Networks [hn, 3 points, 0 comments]

Related

• Sobolev acceleration for neural networks
• Global Convergence of Sobolev Training for Overparameterized Neural Networks
• Sobolev Training for Operator Learning
• Sobolev Training for Implicit Neural Representations with Approximated Image Derivatives
• Approximation Power of Deep Neural Networks: an explanatory mathematical survey