A tutorial on conformal prediction

2007/06/21 by Glenn Shafer, Vladimir Vovk · 14 citations
Computer Science · Mathematics · #cs.LG #stat.ML

paper · pdf

published as Journal of Machine Learning Research 9 (2008) 371-421. http://www.jmlr.org/papers/v9/shafer08a.html · 58 pages, 9 figures

arxiv created 2007/06/21 · arxiv updated 2019/04/23

Abstract

Conformal prediction uses past experience to determine precise levels of confidence in new predictions. Given an error probability ε, together with a method that makes a prediction y of a label y, it produces a set of labels, typically containing y, that also contains y with probability 1-ε. Conformal prediction can be applied to any method for producing y: a nearest-neighbor method, a support-vector machine, ridge regression, etc. Conformal prediction is designed for an on-line setting in which labels are predicted successively, each one being revealed before the next is predicted. The most novel and valuable feature of conformal prediction is that if the successive examples are sampled independently from the same distribution, then the successive predictions will be right 1-ε of the time, even though they are based on an accumulating dataset rather than on independent datasets. In addition to the model under which successive examples are sampled independently, other on-line compression models can also use conformal prediction. The widely used Gaussian linear model is one of these. This tutorial presents a self-contained account of the theory of conformal prediction and works through several numerical examples. A more comprehensive treatment of the topic is provided in "Algorithmic Learning in a Random World", by Vladimir Vovk, Alex Gammerman, and Glenn Shafer (Springer, 2005).

Cited by

• Online Reasoning Calibration: Test-Time Training Enables Generalizable Conformal LLM Reasoning
• Report the Floor: A Training-Free Conformal Interval Is a Mandatory Baseline for Probabilistic Time-Series Forecasting
• LoBoost: Fast Model-Native Local Conformal Prediction for Gradient-Boosted Trees
• A Deployment Audit of Release-Side Risk in Conformal Triage under Prevalence Shift
• Training-Free Probabilistic Time-Series Forecasting with Conformal Seasonal Pools
• Aerial Image Classification in Scarce and Unconstrained Environments via Conformal Prediction
• Enhancing gravitational-wave detection: a machine learning pipeline combination approach with robust uncertainty quantification
• GUARD: Grounding Uncertainty and Ablation-Based Risk Detection for Diffusion-Based VLAs
• ConformalNL2LTL: Translating Natural Language Instructions into Temporal Logic Formulas with Conformal Correctness Guarantees
• From predictions to confidence intervals: an empirical study of conformal prediction methods for in-context learning
• Fair Conformal Prediction for Incomplete Covariate Data
• Efficient MAP Estimation of LLM Judgment Performance with Prior Transfer
• Matrix effects in laser-induced breakdown spectroscopy: A review from fundamental mechanisms to data-driven modeling strategies
• FAIR-SIGHT: Fairness Assurance in Image Recognition via Simultaneous Conformal Thresholding and Dynamic Output Repair

Related

• Elements of Conformal Prediction for Statisticians
• On some practical challenges of conformal prediction
• Theoretical Foundations of Conformal Prediction
• Unifying Different Theories of Conformal Prediction
• Conformal Prediction: A Data Perspective