A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification
2021/07/15 by Anastasios N. Angelopoulos, Stephen Bates, Angelopoulos, Anastasios N. +1 · 3 voices · 220 citations
Computer Science · Mathematics · #Adversarial Robustness in Machine Learning #Anomaly Detection Techniques and Applications #Artificial intelligence #Computer science #Explainable Artificial Intelligence (XAI) #Free probability #Machine learning #Mathematics #Probability distribution #Programming language #Python (programming language) #Statistician #Statistics #Syntax #Theoretical computer science #Uncertainty quantification
paper · pdf · doi:10.48550/arxiv.2107.07511
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2021/07/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
Abstract
Black-box machine learning models are now routinely used in high-risk settings, like medical diagnostics, which demand uncertainty quantification to avoid consequential model failures. Conformal prediction is a user-friendly paradigm for creating statistically rigorous uncertainty sets/intervals for the predictions of such models. Critically, the sets are valid in a distribution-free sense: they possess explicit, non-asymptotic guarantees even without distributional assumptions or model assumptions. One can use conformal prediction with any pre-trained model, such as a neural network, to produce sets that are guaranteed to contain the ground truth with a user-specified probability, such as 90%. It is easy-to-understand, easy-to-use, and general, applying naturally to problems arising in the fields of computer vision, natural language processing, deep reinforcement learning, and so on. This hands-on introduction is aimed to provide the reader a working understanding of conformal prediction and related distribution-free uncertainty quantification techniques with one self-contained document. We lead the reader through practical theory for and examples of conformal prediction and describe its extensions to complex machine learning tasks involving structured outputs, distribution shift, time-series, outliers, models that abstain, and more. Throughout, there are many explanatory illustrations, examples, and code samples in Python. With each code sample comes a Jupyter notebook implementing the method on a real-data example; the notebooks can be accessed and easily run using our codebase.
Citations
Cited by
- Combining gravitational wave search pipelines to find subthreshold signals in GWTC-5.0
- pacSTL: PAC-Bounded Signal Temporal Logic from Data-Driven Reachability Analysis
- When to Trust the Map: Confidence-Aware LLM Routing for Automotive CVE-to-ATM Mapping
- AEVAL: From Anecdotal to Deterministic Testing for Agentic Skill Workflows
- Signed Evidence Flow: Conflict-Aware and Stability-Calibrated Data Analysis
- Falsifiable Release Gates for Self-Improving Systems: Standing Invariants at Scale
- Heteroskedastic Signals in Budgeted LLM Verification: Structural Heterogeneity Limits Optimization Gains
- Multi-Granularity Conformal Prediction for Reliable Neural-Operator Automotive Aerodynamic Surrogates
- Evaluating Conformal Reliability of Pathway-Level Transcriptomic Signatures Under Cross-Cohort Shift in Sepsis Mortality Prediction
- Spectral Adaptive Conformal Prediction for Structured Non-Exchangeable Data
- Contrastive Conformal Sets
- Coverage Guarantees for Pseudo-Calibrated Conformal Prediction under Distribution Shift
- Drift-Aware Spectral Conformal Prediction for Non-Exchangeable Streaming Data
- A Step Forward Towards Trustworthy Risk-Aware Facial Retrieval (RA-FR)
- When Is Delegated Play Truthful? Within-Range Regret and the Trilemma of Aligned Delegation
- Cloud-Native Evaluation-as-a-Service: A Microservices Architecture for Scalable AI Monitoring with Conformal Guarantees
- A Collectivist, Economic Perspective on AI
- Distribution-Free Process Monitoring with Conformal Prediction
- proxymate: Diagnosis and Adjustment of Proxy Estimates for Reliable Inference
- Conformal predictive intervals in survival analysis: a re-sampling approach
- Vehicle-to-Everything Cooperative Perception for Autonomous Driving
- FinAbstain: Uncertainty-Calibrated Multimodal RAG for Selective Financial Forecasting
- Gradient-Based Latent Decomposition Reveals Mechanisms of Feature Degradation in Weakly Supervised Mammography
- A Diagnostic Gap Framework for Evaluating Reconstruction Fidelity in Weakly Supervised Mammography
- Efficient Online Conformal Selection with Limited Feedback
- Enhancing Decision-Making in Windows PE Malware Classification During Dataset Shifts with Uncertainty Estimation
- Another Fit Bites the Dust: Conformal Prediction as a Calibration Standard for Machine Learning in High-Energy Physics
- The Semantic Illusion: Certified Limits of Embedding-Based Hallucination Detection in RAG Systems
- Weighted Conformal Prediction Provides Adaptive and Valid Mask-Conditional Coverage for General Missing Data Mechanisms
- General OOD Detection via Model-aware and Subspace-aware Variable Priority
- Selective Conformal Risk Control
- LYNX: Learning Dynamic Exits for Confidence-Controlled Reasoning
- Seeing Soil from Space: Towards Robust and Scalable Remote Soil Nutrient Analysis
- A Critical Perspective on Finite Sample Conformal Prediction Theory in Medical Applications
- Symmetric Aggregation of Conformity Scores for Efficient Uncertainty Sets
- Posterior inference via Hill's prediction model
- Reliable Statistical Guarantees for Conformal Predictors with Small Datasets
- Pick-to-Learn for Systems and Control: Data-driven Synthesis with State-of-the-art Safety Guarantees
- Modal Logical Neural Networks
- LEC: Linear Expectation Constraints for Selection-Conditioned Risk Control in Selective Prediction and Routing Systems
- What AI Speaks for Your Community: Polling AI Agents for Public Opinion on Data Center Projects
- Sample-Efficient Expert Query Control in Active Imitation Learning via Conformal Prediction
- Beyond Accuracy: An Empirical Study of Uncertainty Estimation in Imputation
- Conformal Safety Monitoring for Flight Testing: A Case Study in Data-Driven Safety Learning
- Safe and Stable Neural Network Dynamical Systems for Robot Motion Planning
- Enhancing Conformal Prediction via Class Similarity
- Hierarchical biomarker thresholding: a model-agnostic framework for stability
- Conformal Prediction for Compositional Data
- Principled Context Engineering for RAG: Statistical Guarantees via Conformal Prediction
- Cost-Sensitive Conformal Training with Provably Controllable Learning Bounds
- Provably Minimum-Length Conformal Prediction Sets for Ordinal Classification
- Attention-Based Feature Online Conformal Prediction for Time Series
- Controlling False Positives in Image Segmentation via Conformal Prediction
- Conformal Online Learning of Deep Koopman Linear Embeddings
- Calibrated Decomposition of Aleatoric and Epistemic Uncertainty in Deep Features for Inference-Time Adaptation
- Safe Planning in Interactive Environments via Iterative Policy Updates and Adversarially Robust Conformal Prediction
- Semi-Supervised Treatment Effect Estimation with Unlabeled Covariates for Prediction-Powered Causal Inference
- Statistically Assuring Safety of Control Systems using Ensembles of Safety Filters and Conformal Prediction
- CoFineLLM: Conformal Finetuning of LLMs for Language-Instructed Robot Planning
- Uncertainty Quantification for Reduced-Order Surrogate Models Applied to Cloud Microphysics
- Conformal Prediction-Driven Adaptive Sampling for Digital Twins of Water Distribution Networks
- Relational Deep Dive: Error-Aware Queries Over Unstructured Data
- Online Energy Storage Arbitrage under Imperfect Predictions: A Conformal Risk-Aware Approach
- Dynamic Model Selection for Trajectory Prediction via Pairwise Ranking and Meta-Features
- Before the Clinic: Transparent and Operable Design Principles for Healthcare AI
- A systematic evaluation of uncertainty quantification techniques in deep learning: a case study in photoplethysmography signal analysis
- Simultaneous Coverage and Efficiency Guarantee in Online Conformal Prediction
- Towards Trustworthy Embodied Intelligence: A Systems Framework and Graded Trustworthiness Levels
- Structurally Separated Uncertainty in Supervised Latent Variable Models
- LRT-Diffusion: Calibrated Risk-Aware Guidance for Diffusion Policies
- Conformal Prediction in The Loop: A Feedback-Based Uncertainty Model for Trajectory Optimization
- Robust Uncertainty Quantification for Self-Evolving Large Language Models via Continual Domain Pretraining
- Multi-Modal Fact-Verification Framework for Reducing Hallucinations in Large Language Models
- What Does It Take to Build a Performant Selective Classifier?
- Foundation Model Forecasts: Form and Function
- Extending Prediction-Powered Inference through Conformal Prediction
- Adaptive Individual Uncertainty under Out-Of-Distribution Shift with Expert-Routed Conformal Prediction
- Federated Conditional Conformal Prediction via Generative Models
- Km-scale dynamical downscaling through conformalized latent diffusion models
- UNCAP: Uncertainty-Guided Planning Using Natural Language Communication for Cooperative Autonomous Vehicles
- Simultaneous Frequentist Calibration of Confidence Regions for Multiple Functionals in Constrained Inverse Problems
- Uncertainty-Aware, Risk-Adaptive Access Control for Agentic Systems using an LLM-Judged TBAC Model
- Modelling Territorial Dynamics through Statistical Mechanics: An Application to Resident Foreign Population
- SAFE: Multitask Failure Detection for Vision-Language-Action Models
- metabeta -- A fast neural model for Bayesian mixed-effects regression
- Valid Stopping for LLM Generation via Empirical Dynamic Formal Lift
- INSIGHT: INference-time Sequence Introspection for Generating Help Triggers in Vision-Language-Action Models
- Domain-Shift-Aware Conformal Prediction for Large Language Models
- Uncertainty in Machine Learning
- Relevance-Aware Thresholding in Online Conformal Prediction for Time Series
- Annotation-Efficient Active Test-Time Adaptation with Conformal Prediction
- Safe Planning in Unknown Environments Using Conformalized Semantic Maps
- Uncertainty-Guided Expert-AI Collaboration for Efficient Soil Horizon Annotation
- Conformal Prediction for Signal Temporal Logic Inference
- Uncertainty quantification in atomistic simulations of silicon using interatomic potentials
- Taming Variability: Randomized and Bootstrapped Conformal Risk Control for LLMs
- Structure and transport properties of LiTFSI-based deep eutectic electrolytes from machine-learned interatomic potential simulations
- Smoothing-Based Conformal Prediction for Balancing Efficiency and Interpretability
- COMPASS: Robust Feature Conformal Prediction for Medical Segmentation Metrics
- Unsupervised Conformal Inference: Bootstrapping and Alignment to Control LLM Uncertainty
- Conditional predictive inference with Lk-coverage control
- A Systematic Review of Conformal Inference Procedures for Treatment Effect Estimation: Methods and Challenges
- Filtering with Confidence: When Data Augmentation Meets Conformal Prediction
- Document Summarization with Conformal Importance Guarantees
- ConRad: Efficient Conformal Prediction for Radiomics
- Geometric Risk Control for Vision-Language Model OCR
- Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Prediction
- Are you sure? Measuring models bias in content moderation through uncertainty
- A Conformal Prediction Framework for Uncertainty Quantification in Physics-Informed Neural Networks
- The Art of Saying "Maybe": A Conformal Lens for Uncertainty Benchmarking in VLMs
- No Need for Learning to Defer? A Training Free Deferral Framework to Multiple Experts through Conformal Prediction
- Selective Risk Certification for LLM Outputs via Information-Lift Statistics: PAC-Bayes, Robustness, and Skeleton Design
- Probabilistic Robustness Analysis in High Dimensional Space: Application to Semantic Segmentation Network
- Feasibility-Guided Fair Adaptive Offline Reinforcement Learning for Medicaid Care Management
- Measuring Uncertainty in Transformer Circuits with Effective Information Consistency
- Statistical Methods in Generative AI
- Performance of Conformal Prediction in Capturing Aleatoric Uncertainty
- Fundamental bounds on efficiency-confidence trade-off for transductive conformal prediction
- Conformal Prediction for Time-series Forecasting with Change Points
- Towards Trustworthy Vital Sign Forecasting: Leveraging Uncertainty for Prediction Intervals
- Energy Landscapes Enable Reliable Abstention in Retrieval-Augmented Large Language Models for Healthcare
- CUTE-MRI: Conformalized Uncertainty-based framework for Time-adaptivE MRI
- Effect of Data Augmentation on Conformal Prediction for Diabetic Retinopathy
- TCUQ: Single-Pass Uncertainty Quantification from Temporal Consistency with Streaming Conformal Calibration for TinyML
- Uncertainty-Aware Prediction of Parkinson's Disease Medication Needs: A Two-Stage Conformal Prediction Approach
- Scalable, Technology-Agnostic Diagnosis and Predictive Maintenance for Point Machine using Deep Learning
- CVCM Track Circuits Pre-emptive Failure Diagnostics for Predictive Maintenance Using Deep Neural Networks
- Membership Inference Attacks with False Discovery Rate Control
- Conformal P-Value in Multiple-Choice Question Answering Tasks with Provable Risk Control
- Conformal Sets in Multiple-Choice Question Answering under Black-Box Settings with Provable Coverage Guarantees
- Conformal Unlearning: A New Paradigm for Unlearning in Conformal Predictors
- Random-Access Ranked Retrieval and Similarity Search
- ConformalSAM: Unlocking the Potential of Foundational Segmentation Models in Semi-Supervised Semantic Segmentation with Conformal Prediction
- Safety Evaluation of Motion Plans Using Trajectory Predictors as Forward Reachable Set Estimators
- Multi-Agent Path Finding Among Dynamic Uncontrollable Agents with Statistical Safety Guarantees
- Approximating Full Conformal Prediction for Neural Network Regression with Gauss-Newton Influence
- Programmable Virtual Humans Toward Human Physiologically-Based Drug Discovery
- Auto: The AGI Compiler
- QUTCC: Quantile Uncertainty Training and Conformal Calibration for Imaging Inverse Problems
- AEGIS: A Backup Reflex for Physical AI
- Heterogeneous Predictor-based Risk-Aware Planning with Conformal Prediction in Dense, Uncertain Environments
- Differentially Private Conformal Prediction via Quantile Binary Search
- LLM-based ambiguity detection in natural language instructions for collaborative surgical robots
- Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control
- Non-exchangeable Conformal Prediction with Optimal Transport: Tackling Distribution Shifts with Unlabeled Data
- Foundation models for time series forecasting: Application in conformal prediction
- Conformal Prediction for Long-Tailed Classification
- Machine Learning in Acoustics: A Review and Open-Source Repository
- WitCert: Sound Runtime Risk Observability and Gating for KV-Cache Quantization
- MARGIN: Runtime Confidence Calibration for Multi-Agent Foundation Model Coordination
- Uncertainty-Aware Complex Scientific Table Data Extraction
- Meteorologically-Informed Adaptive Conformal Prediction for Tropical Cyclone Intensity Forecasting
- Response Quality Assessment for Retrieval-Augmented Generation via Conditional Conformal Factuality
- Pitfalls of Conformal Predictions for Medical Image Classification
- Unveiling Complex Territorial Socio-Economic Dynamics: A Statistical Mechanics Approach
- Profile monitoring of random functions with Gaussian process basis expansions
- MARCO: Hardware-Aware Neural Architecture Search for Edge Devices with Multi-Agent Reinforcement Learning and Conformal Prediction Filtering
- CertDW: Towards Certified Dataset Ownership Verification via Conformal Prediction
- Private List Learnability vs. Online List Learnability
- Conformal Safety Shielding for Imperfect-Perception Agents
- Direct Prediction Set Minimization via Bilevel Conformal Classifier Training
- Statistical Guarantees in Data-Driven Nonlinear Control: Conformal Robustness for Stability and Safety
- Conformal Prediction Adaptive to Unknown Subpopulation Shifts
- AmbiK: Dataset of Ambiguous Tasks in Kitchen Environment
- PARC: A Quantitative Framework Uncovering the Symmetries within Vision Language Models
- Conformal Arbitrage: Risk-Controlled Balancing of Competing Objectives in Language Models
- Performative Risk Control: Calibrating Models for Reliable Deployment under Performativity
- Conformal Object Detection by Sequential Risk Control
- JAPAN: Joint Adaptive Prediction Areas with Normalising-Flows
- Smart Surrogate Losses for Contextual Stochastic Linear Optimization with Robust Constraints
- Deep Learning-Based BMD Estimation from Radiographs with Conformal Uncertainty Quantification
- Risk-Sensitive Conformal Prediction for Catheter Placement Detection in Chest X-rays
- Semi-Supervised Conformal Prediction With Unlabeled Nonconformity Score
- CARE: Confidence-aware Ratio Estimation for Medical Biomarkers
- CP-Router: An Uncertainty-Aware Router Between LLM and LRM
- MetaSTNet: Multimodal Meta-learning for Cellular Traffic Conformal Prediction
- Conformal Prediction for Uncertainty Estimation in Drug-Target Interaction Prediction
- Trust, or Don't Predict: Introducing the CWSA Family for Confidence-Aware Model Evaluation
- Thought calibration: Efficient and confident test-time scaling
- Bayesian Deep Learning for Discrete Choice
- A Generic Framework for Conformal Fairness
- Multivariate Latent Recalibration for Conditional Normalizing Flows
- Conformal Language Model Reasoning with Coherent Factuality
- Phasebook: A Survey of Selected Open Problems in Phase Retrieval
- Domain Adaptive Skin Lesion Classification via Conformal Ensemble of Vision Transformers
- Know When to Abstain: Optimal Selective Classification with Likelihood Ratios
- Generative AI for Autonomous Driving: A Review
- Finding Distributions that Differ, with False Discovery Rate Control
- Synthetic-Powered Predictive Inference
- When Retrieval Helps and Distracts: Evaluating Evidence-Generating LLMs for Biomedical Claim Verification
- Conformalized Large Language Models under Configuration Shift
- ST-LoRA: Single Trajectory LoRA Ensemble for Uncertainty Aware Agricultural Segmentation
- Conformalized Decision Risk Assessment
- Validation of Conformal Prediction in Cervical Atypia Classification
- Conformal Prediction for Verifiable Learned Query Optimization
- Uncertainty Quantification for Machine Learning in Healthcare: A Survey
- Online Reasoning Calibration: Test-Time Training Enables Generalizable Conformal LLM Reasoning
- Fast LLM-Based Semantic Filtering: From a Unified Framework to an Adaptive Two-Phase Method
- Uncertainty-Aware Solar Flare Regression
- Scalable Uncertainty Quantification for Extreme Weather Forecasting via Empirical Neural Tangent Kernels
- Foundations of a Time-Consistent Counterfactual Actuarial Runtime for Autonomous AI Agents
- Insuring Every Action: An Authority Frontier Framework for Runtime Actuarial Control of Autonomous AI Agents
- An empirical evaluation of the risks of AI model updates using clinical data: stability, arbitrariness, and fairness
- Calibrating Conservatism for Scalable Oversight
- Neural posterior estimation for scalable and accurate inverse parameter inference in Li-ion batteries
- CorePath: A Breast-Specialized Pathology Foundation Model for Core Needle Biopsy Diagnosis and Risk-Controlled Report Generation
- Beyond Predicting Responses: Conformal Inference for Latent Distributional Parameters
- Conformal risk control for model-form uncertainty in parametric non-intrusive reduced-order models
- Plug-and-Play Physics-informed Learning using Uncertainty Quantified Port-Hamiltonian Models
- Enhancing gravitational-wave detection: a machine learning pipeline combination approach with robust uncertainty quantification
- GUARD: Grounding Uncertainty and Ablation-Based Risk Detection for Diffusion-Based VLAs
- SAFECAST: Robust Failure Detection for VLA Policies with Contrast-Set Training and Calibration
- Statistical Management of the False Discovery Rate in Medical Instance Segmentation Based on Conformal Risk Control
- ConformalNL2LTL: Translating Natural Language Instructions into Temporal Logic Formulas with Conformal Correctness Guarantees
- Conformalized-KANs: Uncertainty Quantification with Coverage Guarantees for Kolmogorov-Arnold Networks (KANs) in Scientific Machine Learning
- Leave-One-Out Stable Conformal Prediction
- SkillTFM: Gated Skill Evolution for Training-Free Adaptation of Tabular Foundation Models
- Evidential Rule Learning for Interpretable Classification with Abstention
- Trustworthy AI Must Account for Interactions
- Conformal Robust Beamforming via Generative Channel Models
Discussions
- Ever run a complicated supervised learning alg., need to quantify uncertainty, but have no idea how to calculate standard errors?
Conformal prediction provides a simple, versatile, finite-sample-valid [bsky, 26 points, 1 comments]
- little known fact, you can train a model to have a specific FNR ("conformal risk control") arxiv.org/abs/2107.07511 [bsky, 9 points, 0 comments]
- I looked it up, this is a nice paper on it: arxiv.org/pdf/2107.07511 [bsky, 0 points, 0 comments]
Related