Learning Surrogate Losses

2019/05/24 by Grabocka, Josif, Scholz, Randolf, Schmidt-Thieme, Lars
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.1905.10108

Abstract

The minimization of loss functions is the heart and soul of Machine Learning. In this paper, we propose an off-the-shelf optimization approach that can minimize virtually any non-differentiable and non-decomposable loss function (e.g. Miss-classification Rate, AUC, F1, Jaccard Index, Mathew Correlation Coefficient, etc.) seamlessly. Our strategy learns smooth relaxation versions of the true losses by approximating them through a surrogate neural network. The proposed loss networks are set-wise models which are invariant to the order of mini-batch instances. Ultimately, the surrogate losses are learned jointly with the prediction model via bilevel optimization. Empirical results on multiple datasets with diverse real-life loss functions compared with state-of-the-art baselines demonstrate the efficiency of learning surrogate losses.

Related

• Relational Surrogate Loss Learning
• A Unified Framework of Surrogate Loss by Refactoring and Interpolation
• Novel and Efficient Approximations for Zero-One Loss of Linear Classifiers
• A Universal Growth Rate for Learning with Smooth Surrogate Losses
• Learning Differentiable Surrogate Losses for Structured Prediction