Imposing Hard Constraints on Deep Networks: Promises and Limitations

2017/06/07 by Pablo Márquez-Neila, Mathieu Salzmann, Márquez-Neila, Pablo +3 · 16 citations
Computer Science · #Constraint Satisfaction and Optimization #Human Pose and Action Recognition #Neural Networks and Applications #cs.CV

paper · pdf · doi:10.48550/arxiv.1706.02025

arxiv created 2017/06/07 · arxiv updated 2017/06/08

Abstract

Imposing constraints on the output of a Deep Neural Net is one way to improve the quality of its predictions while loosening the requirements for labeled training data. Such constraints are usually imposed as soft constraints by adding new terms to the loss function that is minimized during training. An alternative is to impose them as hard constraints, which has a number of theoretical benefits but has not been explored so far due to the perceived intractability of the problem. In this paper, we show that imposing hard constraints can in fact be done in a computationally feasible way and delivers reasonable results. However, the theoretical benefits do not materialize and the resulting technique is no better than existing ones relying on soft constraints. We analyze the reasons for this and hope to spur other researchers into proposing better solutions.

Citations

• ADADELTA: An Adaptive Learning Rate Method

Cited by

• A Multi-stage Constrained Optimization Framework for Data-driven Problems
• DAE-HardNet: A Physics Constrained Neural Network Enforcing Differential-Algebraic Hard Constraints
• One Size Fits All: Can We Train One Denoiser for All Noise Levels?
• Homotopy-Guided Self-Supervised Learning of Parametric Solutions for AC Optimal Power Flow
• Constrained Performance Boosting Control for Nonlinear Systems
• A Semantic Loss Function for Deep Learning with Symbolic Knowledge
• Physics-Informed Neural Network Methods for Predicting Plant Height Development
• BEKAN: Boundary condition-guaranteed evolutionary Kolmogorov-Arnold networks with radial basis functions for solving PDE problems
• Constrained Deep Learning using Conditional Gradient and Applications in Computer Vision
• Whole-body Motion Control of an Omnidirectional Wheel-Legged Mobile Manipulator via Contact-Aware Dynamic Optimization
• Pinet: Optimizing hard-constrained neural networks with orthogonal projection layers
• Physics-Informed Neural Networks with Hard Nonlinear Equality and Inequality Constraints
• Heterogeneous Causal Learning for Optimizing Aggregated Functions in User Growth
• Improving Deep Learning Models via Constraint-Based Domain Knowledge: a Brief Survey
• Deep learning calibration of option pricing models: some pitfalls and solutions
• Deep Declarative Networks

Related

• HardNet: Hard-Constrained Neural Networks with Universal Approximation Guarantees
• Constrained Empirical Risk Minimization: Theory and Practice
• DeepSaDe: Learning Neural Networks that Guarantee Domain Constraint Satisfaction
• Enforcing Hard Linear Constraints in Deep Learning Models with Decision Rules
• Neural Fields with Hard Constraints of Arbitrary Differential Order