Coverage-Constrained Human-AI Cooperation with Multiple Experts

2024/11/18 by Zheng Zhang, Zhang, Zheng, Cuong Nguyen +10 · 1 voice · 3 citations
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multi-Agent Systems and Negotiation #cs.CV #cs.LG

paper · pdf · doi:10.48550/arxiv.2411.11976

openalex publication_date 2024/11/18 · arxiv published 2024/11/18 · arxiv updated 2024/12/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Human-AI cooperative classification (HAI-CC) approaches aim to develop hybrid intelligent systems that enhance decision-making in various high-stakes real-world scenarios by leveraging both human expertise and AI capabilities. Current HAI-CC methods primarily focus on learning-to-defer (L2D), where decisions are deferred to human experts, and learning-to-complement (L2C), where AI and human experts make predictions cooperatively. However, a notable research gap remains in effectively exploring both L2D and L2C under diverse expert knowledge to improve decision-making, particularly when constrained by the cooperation cost required to achieve a target probability for AI-only selection (i.e., coverage). In this paper, we address this research gap by proposing the Coverage-constrained Learning to Defer and Complement with Specific Experts (CL2DC) method. CL2DC makes final decisions through either AI prediction alone or by deferring to or complementing a specific expert, depending on the input data. Furthermore, we propose a coverage-constrained optimisation to control the cooperation cost, ensuring it approximates a target probability for AI-only selection. This approach enables an effective assessment of system performance within a specified budget. Also, CL2DC is designed to address scenarios where training sets contain multiple noisy-label annotations without any clean-label references. Comprehensive evaluations on both synthetic and real-world datasets demonstrate that CL2DC achieves superior performance compared to state-of-the-art HAI-CC methods.

Cited by

• Budgeted Multiple-Expert Deferral
• Achieving Trustworthy Real-Time Decision Support Systems with Low-Latency Interpretable AI Models
• DeCoDe: Defer-and-Complement Decision-Making via Decoupled Concept Bottleneck Models

Discussions

• Congratulate Zheng for his AAAI 2026 paper "Coverage-constrained human-AI cooperation with multiple experts" (arxiv.org/abs/2411.11976) that frames the human-AI collaboration as a constrained optimisa [bsky, 1 points, 0 comments]

Related

• Learning to Complement and to Defer to Multiple Users
• Human-AI Collaboration in Decision-Making: Beyond Learning to Defer
• Forming Effective Human-AI Teams: Building Machine Learning Models that Complement the Capabilities of Multiple Experts
• Learning Complementary Policies for Human-AI Teams
• DeCoDe: Defer-and-Complement Decision-Making via Decoupled Concept Bottleneck Models