Selection by Prediction with Conformal p-values

2022/10/04 by Ying Jin, Emmanuel J. Candès, Jin, Ying +1 · 13 citations
Computer Science · Mathematics · #FOS: Computer and information sciences #Machine Learning (stat.ML) #Machine Learning and Algorithms #Machine Learning and Data Classification #Methodology (stat.ME) #Statistical Methods in Clinical Trials

paper · pdf · doi:10.48550/arxiv.2210.01408

openalex publication_date 2022/10/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Decision making or scientific discovery pipelines such as job hiring and drug discovery often involve multiple stages: before any resource-intensive step, there is often an initial screening that uses predictions from a machine learning model to shortlist a few candidates from a large pool. We study screening procedures that aim to select candidates whose unobserved outcomes exceed user-specified values. We develop a method that wraps around any prediction model to produce a subset of candidates while controlling the proportion of falsely selected units. Building upon the conformal inference framework, our method first constructs p-values that quantify the statistical evidence for large outcomes; it then determines the shortlist by comparing the p-values to a threshold introduced in the multiple testing literature. In many cases, the procedure selects candidates whose predictions are above a data-dependent threshold. Our theoretical guarantee holds under mild exchangeability conditions on the samples, generalizing existing results on multiple conformal p-values. We demonstrate the empirical performance of our method via simulations, and apply it to job hiring and drug discovery datasets.

Cited by

• Distribution-Free Selection of Low-Risk Oncology Patients for Survival Beyond a Time Horizon
• Asymptotic Uniform False Discovery Rate Control for Inference of Time-varying Correlations
• Symmetric Aggregation of Conformity Scores for Efficient Uncertainty Sets
• LEC: Linear Expectation Constraints for Selection-Conditioned Risk Control in Selective Prediction and Routing Systems
• Elements of Conformal Prediction for Statisticians
• A Bayesian Model for Multi-stage Censoring
• Extending Prediction-Powered Inference through Conformal Prediction
• Selective Labeling with False Discovery Rate Control
• Provable Training Data Identification for Large Language Models
• Theoretical guarantees for change localization using conformal p-values
• Multi-Condition Conformal Selection
• ATTS: Asynchronous Test-Time Scaling via Conformal Prediction
• Online Conformal Selection with Accept-to-Reject Changes

Related

• Confidence on the Focal: Conformal Prediction with Selection-Conditional Coverage
• Optimized Conformal Selection: Powerful Selective Inference After Conformity Score Optimization
• Model-free selective inference under covariate shift via weighted conformal p-values
• Efficient Conformal Prediction via Cascaded Inference with Expanded Admission
• Robust Conformalized Selection with Noisy Responses