Principled Approaches for Learning to Defer with Multiple Experts

2023/10/23 by Mao, Anqi, Mohri, Mehryar, Zhong, Yutao · 6 citations
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.2310.14774

Abstract

We present a study of surrogate losses and algorithms for the general problem of learning to defer with multiple experts. We first introduce a new family of surrogate losses specifically tailored for the multiple-expert setting, where the prediction and deferral functions are learned simultaneously. We then prove that these surrogate losses benefit from strong H-consistency bounds. We illustrate the application of our analysis through several examples of practical surrogate losses, for which we give explicit guarantees. These loss functions readily lead to the design of new learning to defer algorithms based on their minimization. While the main focus of this work is a theoretical analysis, we also report the results of several experiments on SVHN and CIFAR-10 datasets.

Cited by

• Principled Algorithms for Optimizing Generalized Metrics in Binary Classification
• Beyond Tsybakov: Model Margin Noise and H-Consistency Bounds
• Budgeted Multiple-Expert Deferral
• Adversarial Robustness in One-Stage Learning-to-Defer
• From Deferral to Learning: Online In-Context Knowledge Distillation for LLM Cascades
• Is Uncertainty Quantification a Viable Alternative to Learned Deferral?

Related

• Mastering Multiple-Expert Routing: Realizable H-Consistency and Strong Guarantees for Learning to Defer
• Regression with Multi-Expert Deferral
• Sample Efficient Learning of Predictors that Complement Humans
• Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensembles
• Theory and Algorithms for Learning with Multi-Class Abstention and Multi-Expert Deferral