Comparing the Areas under Two or More Correlated Receiver Operating Characteristic Curves: A Nonparametric Approach
1988/09/01 by Elizabeth R. DeLong, David M. DeLong, Daniel L. Clarke-Pearson +1 · 186 citations
Mathematics · Environmental Science · #Advanced Statistical Methods and Models #Hydrology and Drought Analysis #Hydrological Forecasting Using AI
paper · doi:10.2307/2531595
Abstract
Methods of evaluating and comparing the performance of diagnostic tests are of increasing importance as new tests are developed and marketed. When a test is based on an observed variable that lies on a continuous or graded scale, an assessment of the overall value of the test can be made through the use of a receiver operating characteristic (ROC) curve. The curve is constructed by varying the cutpoint used to determine which values of the observed variable will be considered abnormal and then plotting the resulting sensitivities against the corresponding false positive rates. When two or more empirical curves are constructed based on tests performed on the same individuals, statistical analysis on differences between curves must take into account the correlated nature of the data. This paper presents a nonparametric approach to the analysis of areas under correlated ROC curves, by using the theory on generalized U-statistics to generate an estimated covariance matrix.
Cited by
- Moral Semantics Survive Machine Translation: Cross-Lingual Evidence from Moral Foundations Corpora
- Time‐Dependent ROC Curves for Censored Survival Data and a Diagnostic Marker
- Image-based consensus molecular subtype (imCMS) classification of colorectal cancer using deep learning
- Analysis of the frequency and spectrum of mutations recognised to cause familial hypercholesterolaemia in routine clinical practice in a UK specialist hospital lipid clinic
- Detection of Left Ventricular Systolic Dysfunction From Electrocardiographic Images
- Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine
- A novel faecal<i>Lachnoclostridium</i>marker for the non-invasive diagnosis of colorectal adenoma and cancer
- Total Serum Bilirubin within 3 Months of Hepatoportoenterostomy Predicts Short-Term Outcomes in Biliary Atresia
- Defining and validating a short form Montreal Cognitive Assessment (s-MoCA) for use in neurodegenerative disease
- Added value of FDG-PET for detection of progressive supranuclear palsy
- The Prostate Health Index Selectively Identifies Clinically Significant Prostate Cancer
- Drivers, latent signatures and parsimonious screening of organophosphate triester mixtures in silver pomfret from Vietnamese coastal waters (2021−2023)
- Coding Algorithms for Defining Comorbidities in ICD-9-CM and ICD-10 Administrative Data
- Visceral Adiposity Index Is Associated With Cardiometabolic Multimorbidity and Improves Risk Prediction: The English Longitudinal Study of Ageing
- Diagnostic performance of serum galactomannan and β-D-glucan for invasive aspergillosis in suspected patients: A meta-analysis
- Psychometric Properties of the Mini-Balance Evaluation Systems Test (Mini-BESTest) in Community-Dwelling Individuals With Chronic Stroke
- “Model for Early Allograft Function” Outperforms “Early Allograft Dysfunction” as a Predictor of Transplant Survival
- Coronary CT Angiography–derived Fractional Flow Reserve: Machine Learning Algorithm versus Computational Fluid Dynamics Modeling
- Locally Advanced Breast Cancer: MR Imaging for Prediction of Response to Neoadjuvant Chemotherapy—Results from ACRIN 6657/I-SPY TRIAL
- Quantified Terminal Ileal Motility during MR Enterography as a Biomarker of Crohn Disease Activity: Prospective Multi-Institution Study
- TACL: Tissue-aware contrastive learning for cross-domain information fusion between histopathology and MRI in knee pathology classification
- The pathophysiologic and prognostic significance of acidosis in severe adult malaria
- Liver fibrosis is associated with an increased risk of non‐fatal myocardial infarction
- 3D multi‐view squeeze‐and‐excitation convolutional neural network for lung nodule classification
- Calibrated Alzheimer's Conversion Risk in Mild Cognitive Impairment: Persistent Homology of Clinical Trajectories with Conformal Guarantees
- Quantifying the complexity of trajectory ensembles with clustering-weighted multivariate multiscale sample entropy
- A Transportable Threshold-Based Framework for Interpretable Classification of Medical Data
- Dataset-Origin Signatures and Shortcut Learning in Screening Mammography AI: A Cross-Dataset Case Study
- Widespread use of invalid statistical tests in biomedical machine learning
- A Knowledge-Injection Framework for Zero-Shot Adaptation of LLMs to Delirium Prediction
- Levels of circulating kidney injury markers and IL-10 identify non-critically ill patients with COVID-19 at risk of death
- Do Large Language Models Know What They Are Capable Of?
- Dual-attention ResNet outperforms transformers in HER2 prediction on DCE-MRI
- Foundation models for electrocardiogram interpretation: clinical implications
- Prediction of incident atrial fibrillation using deep learning, clinical models, and polygenic scores
- Long short-term memory network with adapted attention mechanism for credit risk modeling
- Lung cancer screening with low-dose CT scans using a deep learning approach
- Identification of respiratory microbiota markers in ventilator-associated pneumonia
- What Can Be Enforced? A Theory of Certified Runtime Safety for Tool-Using Agents
- Comparing the Areas Under Two Correlated ROC Curves: Parametric and Non‐Parametric Approaches
- Intersectional Fairness in Vision-Language Models for Medical Image Disease Classification
- Credit Risk Estimation with Non-Financial Features: Evidence from a Synthetic Istanbul Dataset
- Arterial blood pressure during early sepsis and outcome
- On the Rise of FinTechs: Credit Scoring Using Digital Footprints
- Identification of Pediatric Sepsis Subphenotypes for Enhanced Machine Learning Predictive Performance: A Latent Profile Analysis
- Humans incorrectly reject confident accusatory AI judgments
- Empirical Assessment of the Code Comprehension Effort Needed to Attack Programs Protected with Obfuscation
- Hierarchical biomarker thresholding: a model-agnostic framework for stability
- Boundary-Aware Adversarial Filtering for Reliable Diagnosis under Extreme Class Imbalance
- Does machine learning improve prediction accuracy of the Endoscopic Third Ventriculostomy Success Score? A contemporary Hydrocephalus Clinical Research Network cohort study
- From 2D to 3D Without Extra Baggage: Data-Efficient Cancer Detection in Digital Breast Tomosynthesis
- Impact of Exposure Parameters on Deep Learning Models in Chest Radiography and Implications for Deployment
- SPASHT: An image-enhancement method for sparse-view MPI SPECT
- Adapted Foundation Models for Breast MRI Triaging in Contrast-Enhanced and Non-Contrast Enhanced Protocols
- Identification of Separable OTUs for Multinomial Classification in Compositional Data Analysis
- A Prospective Examination of the Predictive Validity of Three Transdiagnostic Assessments of Risk for Suicidal Behavior: Psychache, the Interpersonal Theory of Suicide, and Reasons for Living
- Spatially distributed antecedent rainfall thresholds for landslide occurrence: a multitask machine learning modelling approach
- Satellite Remote Sensing Reveals Climate‐Driven Breeding Failure in Greater Flamingos at a Mediterranean Hypersaline Lake
- Fog Forecasting for Melbourne Airport Using a Bayesian Decision Network
- Administrative healthcare data applied to fracture risk assessment
- Comparing the Performance of Foundation Model Derived Embeddings with Traditional Approaches for Distant Metastasis Prediction in Head and Neck Cancer
- Prediction of synchronous distant metastasis of primary pancreatic ductal adenocarcinoma using the radiomics features derived from 18F-FDG PET and MRI
- Evaluating a predictive model of tyrosine kinase inhibitor therapy failure in a European-type cohort: a step towards population-specific tools
- MR-Transformer: A Vision Transformer-based Deep Learning Model for Total Knee Replacement Prediction Using MRI
- Autoantibody and T cell responses to oxidative post-translationally modified insulin neoantigenic peptides in type 1 diabetes
- Multivariate machine learning models for prediction of pathologic response to neoadjuvant therapy in breast cancer using MRI features: a study using an independent validation set
- The initial molecular response predicts the deep molecular response but not treatment-free remission maintenance in a real-world chronic myeloid leukemia cohort
- Using Explainable AI to Characterize Features in the Mirai Mammographic Breast Cancer Risk Prediction Model
- Exploring the Effect of Sentence Structure Frequency on the Accuracy of a Screener for Adults at Risk of Developmental Language Disorder
- Consensus meta-analysis of genome-wide association studies for Alzheimer’s disease and related dementias
- MRI‐Based Machine Learning for Differentiating Borderline From Malignant Epithelial Ovarian Tumors: A Multicenter Study
- Sample size, power and effect size revisited: simplified and practical approaches in pre-clinical, clinical and laboratory studies
- Risk Prediction Model for Postoperative Acute Kidney Injury in a Broad Surgical Population
- Using Topological Data Analysis for diagnosis pulmonary embolism
- Improving Prognostic Performance in Resectable Pancreatic Ductal Adenocarcinoma using Radiomics and Deep Learning Features Fusion in CT Images
- WHO absolute fracture risk models (FRAX): Do clinical risk factors improve fracture prediction in older women without osteoporosis?
- Self-Supervised Text-Vision Alignment for Automated Brain MRI Abnormality Detection: A Multicenter Study (ALIGN Study)
- Adaptive EEG-based stroke diagnosis with a GRU-TCN classifier and deep Q-learning thresholding
- Cross-National Radiomics Validation Using Mammography to Predict Occult Invasion in Ductal Carcinoma in Situ
- Artificial Intelligence Outcome Prediction in Neonates with Encephalopathy (AI-OPiNE)
- Long-Bone Fracture Detection using Artificial Neural Networks based on Contour Features of X-ray Images
- Determination of the IonizingContinuumin Seyfert Galaxies fromNear Infrared Spectroscopy
- Multi-Label Annotation of Chest Abdomen Pelvis Computed Tomography Text\n Reports Using Deep Learning
- Assessing Monotone Dependence: Area Under the Curve Meets Rank Correlation
- Machine Learning for Early Detection of Meningitis: Stacked Ensemble Learning with EHR Data
- Context-aware deep learning using individualized prior information reduces false positives in disease risk prediction and longitudinal health assessment
- Attention Strongly Modulates Reliability of Neural Responses to Naturalistic Narrative Stimuli
- Evaluation of basal sex hormone levels for activation of the hypothalamic–pituitary–gonadal axis
- Uncertainty Quantification for Retrieval-Augmented Reasoning
- The Fairness of Risk Scores Beyond Classification: Bipartite Ranking and the xAUC Metric
- Distinguishing features of long COVID identified through immune profiling
- Curriculum Learning with Synthetic Data for Enhanced Pulmonary Nodule Detection in Chest Radiographs
- Quantifying uncert-AI-nty: Testing the accuracy of LLMs’ confidence judgments
- Assessing Cultural Bias in Recidivism Risk Tools: Differential Item Functioning and Predictive Validity of the ORAC-PCQ and LS/CMI Among Indigenous and Non-Indigenous Men in Quebec's Correctional System
- On the Role of Calibration in Benchmarking Algorithmic Fairness for Skin Cancer Detection
- Large Language Model-Based Uncertainty-Adjusted Label Extraction for Artificial Intelligence Model Development in Upper Extremity Radiography
- Model Monitoring: A General Framework with an Application to Non-life Insurance Pricing
- TinyViT-Batten: Few-Shot Vision Transformer with Explainable Attention for Early Batten-Disease Detection on Pediatric MRI
- Receiver operating characteristic curve: overview and practical use for clinicians
- Cognitive Biases in Street‐Level Resolutions: The Case of Multiple Claims in Tax Disputes
- IAIA-BL: A Case-based Interpretable Deep Learning Model for Classification of Mass Lesions in Digital Mammography
- Machine Friendly Machine Learning: Interpretation of Computed Tomography Without Image Reconstruction
- Calibrating Verbalized Confidence with Self-Generated Distractors
- Breeding for delayed bolting decelerated the circadian clock in cultivated lettuce
- Seeing Through the Blur: Unlocking Defocus Maps for Deepfake Detection
- Measuring autistic burnout: A psychometric validation of the AASPIRE Autistic Burnout Measure in autistic adults
- Developing a novel reference region for [18F]PI-2620-PET imaging to facilitate the assessment of 4-repeat tauopathies
- Impact of Transfer Learning Using Local Data on Performance of a Deep Learning Model for Screening Mammography
- Deep learning reveals pathology-confirmed neuroimaging signatures in Alzheimer’s, vascular and Lewy body dementias
- An <scp>ANI</scp>‐2 enabled open‐source protocol to estimate ligand strain after docking
- Development and internal validation of the Cleveland Clinic Bleeding Model to predict major bleeding risk at admission in medical inpatients
- Detecting amyloid–β pathology in subjective cognitive decline using plasma and CSF biomarkers
- Multimodal Language Models with Modality-Specific Experts for Financial Forecasting from Interleaved Sequences of Text and Time Series
- Evaluation of a Mammography-based Deep Learning Model for Breast Cancer Risk Prediction in a Triennial Screening Program
- Learning the natural history of human disease with generative transformers
- Differential Expression of ATP6V1D and Its Diagnostic Potential in IgA Nephropathy
- The State of Peptide Detectability in Computational Proteomics and Guidelines for AI Applications
- Evaluating the sample size requirements of tree-based ensemble machine learning techniques for clinical risk prediction
- A scoring system for diagnosing heart failure with preserved ejection fraction based on exercise echocardiography
- Measuring and comparing the accuracy of species distribution models with presence–absence data
- Detecting Positive Selection by Modeling Structure Within Images of Genetic Variation
- RISE: Two‐Stage Rank‐Based Identification of High‐Dimensional Surrogate Markers Applied to Vaccinology
- Development and Validation of a Machine Learning–Based Screening Algorithm to Predict High-Risk Hepatitis C Infection
- Concordant interictal stereoelectroencephalographic high‐frequency oscillations and magnetoencephalography predict better surgical outcomes in focal epilepsy
- AI may help to predict thyroid nodule malignancy based on radiomics features from [18F]FDG PET/CT
- A proof-of-concept investigation into predicting follicular carcinoma on ultrasound using topological data analysis and radiomics
- The combined impact of chronic kidney disease and ulcer severity on incident cardiovascular events in patients with diabetes‐related foot ulceration
- Prediction of heart failure and all-cause mortality using cardiac ultrasomics in patients with breast cancer
- Beyond Traditional Risk Scores: Tackling LS/CMI Offender Misclassifications with Machine Learning
- Utility of Cytokine Biomarkers for the Diagnosis of Pediatric Pyogenic Musculoskeletal Infections
- Detection of Impaired Cerebral Autoregulation Improves by Increasing Arterial Blood Pressure Variability
- Prospective validation of imaging and serum diagnostic biomarkers of steatohepatitis and fibrosis in MASLD: the LITMUS Imaging Study
- A pilot study of magnetic resonance fingerprinting and radiomics analysis in autosomal dominant polycystic kidney disease
- Machine learning-based reclassification of germline variants of unknown significance: The RENOVO algorithm
- Artificial Intelligence Electrocardiogram and Left Ventricular Systolic Dysfunction in Kenya
- Association of Multiple-trait Polygenic Risk Score with Obesity and Cardiometabolic Diseases in Korean Population
- Optimizing sPESI with heart rate threshold adjustments for risk stratification in acute pulmonary embolism: A retrospective cohort study
- Towards Optimal Convolutional Transfer Learning Architectures for Breast Lesion Classification and ACL Tear Detection
- Estimating Semantic Alphabet Size for LLM Uncertainty Quantification
- Conditional Infilling GANs for Data Augmentation in Mammogram Classification
- Accuracy of Respiratory Variation in Inferior Vena Cava Diameter to Predict Fluid Responsiveness in Children Under Mechanical Ventilation
- Combining 2 Commonly Adopted Nutrition Instruments in the Critical Care Setting Is Superior to Administering Either One Alone
- The Effect of Combined Out-of-Hospital Hypotension and Hypoxia on Mortality in Major Traumatic Brain Injury
- Особенности именной префиксации в словообразовательных процессах начала ХХI века
- Furosemide Stress Test and Biomarkers for the Prediction of AKI Severity
- A New Criterion for Pediatric AKI Based on the Reference Change Value of Serum Creatinine
- Improving Performance, Robustness, and Fairness of Radiographic AI Models with Finely-Controllable Synthetic Data
- Do Birthweight‐For‐Gestational Age Centiles Predict Serious Neonatal Morbidity and Neonatal Mortality?
- Comorbidities directly extracted from the hospital database for adjusting SSI risk in the new national semiautomated surveillance system in France: The SPICMI network
- Predicting bankruptcy of local government: A machine learning approach
- Mean arterial pressure values calculated using seven different methods and their associations with target organ deterioration in a single-center study of 1878 individuals
- Trabecular Bone Score (TBS) Predicts Vertebral Fractures in Japanese Women Over 10 Years Independently of Bone Density and Prevalent Vertebral Deformity: The Japanese Population-Based Osteoporosis (JPOS) Cohort Study
- International evaluation of an AI system for breast cancer screening
- Computer algorithms show potential for improving dermatologists' accuracy to diagnose cutaneous melanoma: Results of the International Skin Imaging Collaboration 2017
- 少年事件における犯罪被害者の権利利益の保障(上)少年審判の傍聴制度を中心に
- Towards interpretable prediction of recurrence risk in breast cancer using pathology foundation models
- Street-Level AI: Are Large Language Models Ready for Real-World Judgments?
- Serum uric acid, predicts heart failure in a large Italian cohort: search for a cut-off value the URic acid Right for heArt Health study
- Large-scale Multi-sequence Pretraining for Generalizable MRI Analysis in Versatile Clinical Applications
- FoundBioNet: A Foundation-Based Model for IDH Genotyping of Glioma from Multi-Parametric MRI
- Integrating intestinal ultrasound into routine clinical care improves prediction of biochemical disease activity in pediatric inflammatory bowel disease
- Responsiveness and minimal clinically important difference for pain and disability instruments in low back pain patients. [europepmc]
- Validation of screening questions for limited health literacy in a large VA outpatient population. [europepmc]
- Effectiveness of the AUDIT-C as a screening test for alcohol misuse in three race/ethnic groups. [europepmc]
- Concurrent Validity of the Child Behavior Checklist DSM-Oriented Scales: Correspondence with DSM Diagnoses and Comparison to Syndrome Scales. [europepmc]
- Diffusion-weighted MRI for selection of complete responders after chemoradiation for locally advanced rectal cancer: a multicenter study. [europepmc]
- pROC: an open-source package for R and S+ to analyze and compare ROC curves. [europepmc]
- Non-contrast T1-mapping detects acute myocardial edema with high diagnostic accuracy: a comparison to T2-weighted cardiovascular magnetic resonance. [europepmc]
- Better prediction of functional effects for sequence variants. [europepmc]
- Microbiota-based model improves the sensitivity of fecal immunochemical test for detecting colonic lesions. [europepmc]
- On determining the most appropriate test cut-off value: the case of tests with continuous results. [europepmc]
- Comparison of risk scoring systems for patients presenting with upper gastrointestinal bleeding: international multicentre prospective study. [europepmc]
- Radiomics strategies for risk assessment of tumour failure in head-and-neck cancer. [europepmc]
- Emergency department triage prediction of clinical outcomes using machine learning models. [europepmc]
- Comparison of triglyceride glucose index, and related parameters to predict insulin resistance in Korean adults: An analysis of the 2007-2010 Korean National Health and Nutrition Examination Survey. [europepmc]
- Association of Peritumoral Radiomics With Tumor Biology and Pathologic Response to Preoperative Targeted Therapy for HER2 (ERBB2)-Positive Breast Cancer. [europepmc]
- Performance of Fully Automated Plasma Assays as Screening Tests for Alzheimer Disease-Related β-Amyloid Status. [europepmc]
- Advanced machine learning in action: identification of intracranial hemorrhage on computed tomography scans of the head with clinical workflow integration. [europepmc]
- A novel faecal Lachnoclostridium marker for the non-invasive diagnosis of colorectal adenoma and cancer. [europepmc]
- Discriminative Accuracy of Plasma Phospho-tau217 for Alzheimer Disease vs Other Neurodegenerative Disorders. [europepmc]
- Comprehensive prediction of secondary metabolite structure and biological activity from microbial genome sequences. [europepmc]
- Sample size, power and effect size revisited: simplified and practical approaches in pre-clinical, clinical and laboratory studies. [europepmc]
- Plasma phosphorylated tau 217 and phosphorylated tau 181 as biomarkers in Alzheimer's disease and frontotemporal lobar degeneration: a retrospective diagnostic performance study. [europepmc]
- Biologically informed deep neural network for prostate cancer discovery. [europepmc]
- CheckList for EvaluAtion of Radiomics research (CLEAR): a step-by-step reporting guideline for authors and reviewers endorsed by ESR and EuSoMII. [europepmc]
- Distinguishing features of long COVID identified through immune profiling. [europepmc]
Related