Selective Classification for Deep Neural Networks

2017/05/23 by Yonatan Geifman, Geifman, Yonatan, Ran El‐Yaniv +2 · 1 voice · 65 citations
Computer Science · #Anomaly Detection Techniques and Applications #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification #Neural Networks and Applications #cs.AI #cs.LG

paper · pdf · doi:10.48550/arxiv.1705.08500

openalex publication_date 2017/05/23 · arxiv published 2017/05/23 · arxiv updated 2017/06/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Selective classification techniques (also known as reject option) have not yet been considered in the context of deep neural networks (DNNs). These techniques can potentially significantly improve DNNs prediction performance by trading-off coverage. In this paper we propose a method to construct a selective classifier given a trained neural network. Our method allows a user to set a desired risk level. At test time, the classifier rejects instances as needed, to grant the desired risk (with high probability). Empirical results over CIFAR and ImageNet convincingly demonstrate the viability of our method, which opens up possibilities to operate DNNs in mission-critical applications. For example, using our method an unprecedented 2% error in top-5 ImageNet classification can be guaranteed with probability 99.9%, and almost 60% test coverage.

Citations

• Wide Residual Networks

Cited by

• FinAbstain: Uncertainty-Calibrated Multimodal RAG for Selective Financial Forecasting
• Physically Verifiable Evidence and LLM-Based Reporting for Bearing Fault Diagnosis
• What Can Be Enforced? A Theory of Certified Runtime Safety for Tool-Using Agents
• SAGE: Safety-First Defense-in-Depth Guardrails for Verified Lifecycle Control of High-Impact Generative AI
• Trustworthy Medical Segmentation: Uncertainty-Aware U-Net Evaluation Under Clinical Image Degradation
• Detect Before You Leap: Mirage Detection in Vision-Language Models
• Multi-Layer Confidence Scoring for Detection of Out-of-Distribution Samples, Adversarial Attacks, and In-Distribution Misclassifications
• Can We Test Consciousness Theories on AI? Ablations, Markers, and Robustness
• Interoceptive machine framework: Toward interoception-inspired regulatory architectures in artificial intelligence
• Selective Conformal Risk Control
• Uncertainty Quantification for Machine Learning: One Size Does Not Fit All
• Knowing when to trust machine-learned interatomic potentials
• Detecting AI Hallucinations in Finance: An Information-Theoretic Method Cuts Hallucination Rate by 92%
• ALARM: Automated MLLM-Based Anomaly Detection in Complex-EnviRonment Monitoring with Uncertainty Quantification
• Advancing Image Classification with Discrete Diffusion Classification Modeling
• Honesty over Accuracy: Trustworthy Language Models through Reinforced Hesitation
• Confidence-Aware Neural Decoding of Overt Speech from EEG: Toward Robust Brain-Computer Interfaces
• Catching Contamination Before Generation: Spectral Kill Switches for Agents
• Optimizing Uncertainty-Aware Deep Learning for On-the-Edge Murmur Detection in Low-Resource Settings
• Budgeted Multiple-Expert Deferral
• Towards Explainable and Reliable AI in Finance
• VendorBench-100: A Unified Cross-Paradigm Benchmark for Deepfake Image Detection
• When LLMs Agree, Are They Right? Auditing Self-Consistency and Cross-Model Agreement as Confidence Signals
• BAS: A Decision-Theoretic Approach to Evaluating Large Language Model Confidence
• When Agents Disagree With Themselves: Behavioral Consistency as an Uncertainty Signal for LLM Agents
• GALA: A GlobAl-LocAl Approach for Multi-Source Active Domain Adaptation
• The Gray Zone of Faithfulness: Taming Ambiguity in Unfaithfulness Detection
• Systematic Evaluation of Uncertainty Estimation Methods in Large Language Models
• What Does It Take to Build a Performant Selective Classifier?
• Policy Learning with Abstention
• Evaluating Medical LLMs by Levels of Autonomy: A Survey Moving from Benchmarks to Applications
• Selective Labeling with False Discovery Rate Control
• When In Doubt, Abstain: The Impact of Abstention on Strategic Classification
• Escaping the Agreement Trap: Defensibility Signals for Evaluating Rule-Governed AI
• Learning from AVA: Early Lessons from a Curated and Trustworthy Generative AI for Policy and Development Research
• Valid Stopping for LLM Generation via Empirical Dynamic Formal Lift
• Conservative Decisions with Risk Scores
• Can Molecular Foundation Models Know What They Don't Know? A Simple Remedy with Preference Optimization
• Learning in an Echo Chamber: Online Learning with Replay Adversary
• Limitations on Accurate, Trusted, Human-level Reasoning
• Witness Evidence Portfolios: Single-Prefill Risk Detection for Closed Multimodal Answers
• Rehearse: Stepping Back from the Confidence Cliff in Self-Improving Autoresearch
• When Derived Measurements Mislead: Quantifying and Mitigating LLM Over-Trust with Privileged-Modality Reliability Evidence
• Geometric Risk Control for Vision-Language Model OCR
• LLMs as Signal Detectors: Sensitivity, Bias, and the Temperature-Criterion Analogy
• Similarity-Distance-Magnitude Activations
• Selective Risk Certification for LLM Outputs via Information-Lift Statistics: PAC-Bayes, Robustness, and Skeleton Design
• Know What You Don't Know: Selective Prediction for Early Exit DNNs
• Pseudo-D: Informing Multi-View Uncertainty Estimation with Calibrated Neural Training Dynamics
• TabResFlow: A Normalizing Spline Flow Model for Probabilistic Univariate Tabular Regression
• Predictable Compression Failures: Why Language Models Actually Hallucinate
• Multi-pathology Chest X-ray Classification with Rejection Mechanisms
• Calibrating MLLM-as-a-judge via Multimodal Bayesian Prompt Ensembles
• What Fundamental Structure in Reward Functions Enables Efficient Sparse-Reward Learning?
• Energy Landscapes Enable Reliable Abstention in Retrieval-Augmented Large Language Models for Healthcare
• ALSA: Anchors in Logit Space for Out-of-Distribution Accuracy Estimation
• TCUQ: Single-Pass Uncertainty Quantification from Temporal Consistency with Streaming Conformal Calibration for TinyML
• The Architecture of Trust: A Framework for AI-Augmented Real Estate Valuation in the Era of Structured Data
• Fail Fast, or Ask: Mitigating the Deficiencies of Reasoning LLMs with Human-in-the-Loop Systems Engineering
• Knowing What You Cannot Explain: Learning to Reject Low-Quality Explanations
• Sound and Complete Neurosymbolic Reasoning with LLM-Grounded Interpretations
• Mastering Multiple-Expert Routing: Realizable H-Consistency and Strong Guarantees for Learning to Defer
• Pitfalls of Conformal Predictions for Medical Image Classification
• The Role of Model Confidence on Bias Effects in Measured Uncertainties for Vision-Language Models
• Cascaded Language Models for Cost-effective Human-AI Decision-Making

Discussions

• Selective Classification for Deep Neural Networks [hn, 3 points, 0 comments]

Related

• SelectiveNet: A Deep Neural Network with an Integrated Reject Option
• Hierarchical Selective Classification
• Survey on Leveraging Uncertainty Estimation Towards Trustworthy Deep Neural Networks: The Case of Reject Option and Post-training Processing
• How to Fix a Broken Confidence Estimator: Evaluating Post-hoc Methods for Selective Classification with Deep Neural Networks
• Leveraging Uncertainty in Deep Learning for Selective Classification