Finding Distributions that Differ, with False Discovery Rate Control

2025/05/19 by Lee, Yonghoon, Dobriban, Edgar, Tchetgen, Eric Tchetgen
#FOS: Computer and information sciences #Methodology (stat.ME)

paper · doi:10.48550/arxiv.2505.13769

Abstract

We consider the problem of comparing a reference distribution with several other distributions. Given a sample from both the reference and the comparison groups, we aim to identify the comparison groups whose distributions differ from that of the reference group. Viewing this as a multiple testing problem, we introduce a methodology that provides exact, distribution-free control of the false discovery rate. To do so, we introduce the concept of batch conformal p-values and demonstrate that they satisfy positive regression dependence across the groups [Benjamini and Yekutieli, 2001], thereby enabling control of the false discovery rate through the Benjamini-Hochberg procedure. The proof of positive regression dependence introduces a novel technique for the inductive construction of rank vectors with almost sure dominance under exchangeability. We evaluate the performance of the proposed procedure through simulations, where, despite being distribution-free, in some cases they show performance comparable to methods with knowledge of the data-generating normal distribution; and further have more power than direct approaches based on conformal out-of-distribution detection. Further, we illustrate our methods on a Hepatitis C treatment dataset, where they can identify patient groups with large treatment effects; and on the Current Population Survey dataset, where they can identify sub-population with long work hours.

Citations

• Batch Predictive Inference
• Collective Outlier Detection and Enumeration with Conformalized Closed Testing
• Selection by Prediction with Conformal p-values
• Adaptive novelty detection with false discovery rate guarantee
• A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification
• Comparisons of multiple treatment groups with a negative control or placebo group: Dunnett test vs. closed test procedur
• High-dimensional MANOVA via Bootstrapping and its Application to Functional and Sparse Count Data
• Exchangeability, Conformal Prediction, and Rank Tests
• Minimax optimality of permutation tests
• Conformalized Quantile Regression
• On Wasserstein Two Sample Testing and Related Families of Nonparametric\n Tests
• Adaptive linear step-up procedures that control the false discovery rate
• Permutation Methods: A Basis for Exact Inference
• The control of the false discovery rate in multiple testing under dependency
• Permutation tests for univariate or multivariate analysis of variance and regression
• Table for Estimating the Goodness of Fit of Empirical Distributions
• On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other

Related

• Flexible control of the median of the false discovery proportion
• Controlling the false discovery rate under a non-parametric graphical dependence model
• False Discovery Rate Control For Structured Multiple Testing: Asymmetric Rules And Conformal Q-values
• False Discovery Rate Control Under General Dependence By Symmetrized Data Aggregation
• Model-free selective inference under covariate shift via weighted conformal p-values