E-Valuating Classifier Two-Sample Tests

2022/10/24 by Teodora Pandeva, Pandeva, Teodora, Tim Bakker +5 · 2 citations
Computer Science · Mathematics · #Bayesian Modeling and Causal Inference #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Methodology (stat.ME) #Statistical Methods and Inference

paper · pdf · doi:10.48550/arxiv.2210.13027

openalex publication_date 2022/10/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce a powerful deep classifier two-sample test for high-dimensional data based on E-values, called E-value Classifier Two-Sample Test (E-C2ST). Our test combines ideas from existing work on split likelihood ratio tests and predictive independence tests. The resulting E-values are suitable for anytime-valid sequential two-sample tests. This feature allows for more effective use of data in constructing test statistics. Through simulations and real data applications, we empirically demonstrate that E-C2ST achieves enhanced statistical power by partitioning datasets into multiple batches beyond the conventional two-split (training and testing) approach of standard classifier two-sample tests. This strategy increases the power of the test while keeping the type I error well below the desired significance level.

Cited by

• Keeping Medical AI Healthy and Trustworthy: A Review of Detection and Correction Methods for System Degradation
• On Continuous Monitoring of Risk Violations under Unknown Shift

Related

• Two-Sample Test Based on Classification Probability
• Revisiting Classifier Two-Sample Tests
• A framework for paired-sample hypothesis testing for high-dimensional data
• True and false discoveries with independent and sequential e-values
• Advanced Tutorial: Label-Efficient Two-Sample Tests