The control of the false discovery rate in multiple testing under dependency
2001/08/01 by Yoav Benjamini, Daniel Yekutieli · 10,870 citations
Decision Sciences · Mathematics · #Artificial intelligence #Computer science #Dependency (UML) #False discovery rate #Mathematics #Multiple comparisons problem #Multivariate statistics #Null hypothesis #Optimal Experimental Design Methods #Simple (philosophy) #Statistical Methods and Bayesian Inference #Statistical Methods in Clinical Trials #Statistical hypothesis testing #Statistics #Type I and type II errors
paper · pdf · doi:10.1214/aos/1013699998
published in The Annals of Statistics 29(4) (Institute of Mathematical Statistics)
openalex publication_date 2001/08/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
Abstract
Benjamini and Hochberg suggest that the false discovery rate may be the appropriate error rate to control in many applied multiple testing problems. A simple procedure was given there as an FDR controlling procedure for independent test statistics and was shown to be much more powerful than comparable procedures which control the traditional familywise error rate. We prove that this same procedure also controls the false discovery rate when the test statistics have positive regression dependency on each of the test statistics corresponding to the true null hypotheses. This condition for positive dependency is general enough to cover many problems of practical interest, including the comparisons of many treatments with a single control, multivariate normal test statistics with positive correlation matrix and multivariate t. Furthermore, the test statistics may be discrete, and the tested hypotheses composite without posing special difficulties. For all other forms of dependency, a simple conservative modification of the procedure controls the false discovery rate. Thus the range of problems for which a procedure with proven FDR control can be offered is greatly increased.
Citations
Cited by
- Admissibility and Complete Classes for False Discovery Rate Control with E-values
- Retail Trader's Ruin: An Anatomy of Popular Signal Failure
- E-Values For Multiplicity Control In Multiverse Analysis
- Mirror and knockoff+ thresholds under dependence
- Assessing imbalance in signed brain networks
- A Small network of Consumer-Based wearable sensors Enables The identification of The neutral- and Tilted-Pelvis running techniques
- Longer scans boost prediction and cut costs in brain-wide association studies
- Method: Using generalized additive models in the livestock animal sciences
- A Common Pool of Privacy Problems: Legal and Technical Lessons from a Large-Scale Web-Scraped Machine Learning Dataset
- Climate change drives range contraction and shapes species distribution in an alpine passerine: Caution required when comparing atlas data
- Anatomically resolved oscillatory bursts reveal dynamic motifs of thalamocortical activity during naturalistic stimulus viewing
- Information transfer within and between autistic and non-autistic people
- Improved null proportion estimators for multiple discrete tests with plug-in FDR control
- A General Stability Approach to False Discovery Rate Control
- Efficient and scalable clustering of survival curves
- Asymptotic Uniform False Discovery Rate Control for Inference of Time-varying Correlations
- SUP: An Inferable Private Multiple Testing Framework with Super Uniformity
- Active Hypothesis Testing under Computational Budgets with Applications to GWAS and LLM
- Alignment Faking - the Train -> Deploy Asymmetry: Through a Game-Theoretic Lens with Bayesian-Stackelberg Equilibria
- Elements of Conformal Prediction for Statisticians
- The Probabilistic Foundations of Surveillance Failure: From False Alerts to Structural Bias
- Thinking While Speaking: Inference-Time Knowledge Transfer for Responsive and Intelligent Conversational Voice Agents
- Asymptotics of higher criticism via Gaussian approximation
- Time points or plot points - Are movies processed according to their temporal duration or their underlying content structure?
- Cluster Size Matters: A Comparative Study of Notip and pARI for Post Hoc Inference in fMRI
- Making Interpretable Discoveries from Unstructured Data: A High-Dimensional Multiple Hypothesis Testing Approach
- Hub Neighbor-Degree Diagnostics for Sparse Random Graphs
- Estimation of the complexity of a network under a Gaussian graphical model
- ZW and XY Sex Chromosomes Drive Rapid and Distinctive Evolution of Sex‐Biased Gene Expression
- Selective Labeling with False Discovery Rate Control
- Provable Training Data Identification for Large Language Models
- Multi-Condition Conformal Selection
- Testing the equality of estimable parameters across many populations
- Flexible perceptual encoding by discrete gamma events
- Retrieving the structure of probabilistic sequences from EEG data during the goalkeeper game
- Out-of-bag prediction balls for random forests in metric spaces
- High rates of polygyny do not lock large proportions of men out of the marriage market
- On the Adversarial Robustness of Learning-based Conformal Novelty Detection
- False Discovery Rate Control via Bayesian Mirror Statistic
- Continent-wide stock structure and connectivity of silver trevally (Pseudocaranx georgianus) based on population genomics and otolith shape and chemistry
- Genetic basis of temporal variation in phenology in a woody perennial plant
- Deep Learning–Based Control of Electrically Evoked Activity in Human Visual Cortex
- Genomic connectivity and future climate suitability of the Caribbean king crab (Maguimithrax spinosissimus) highlight its potential for coral reef restoration
- The role of life‐course socioeconomic position in cognitive change and mild cognitive impairment among middle‐aged and older US Hispanic/Latinos
- Genetic Regulation of Anthocyanin Biosynthesis in Cornus Species: The Roles of R2R3‐MYB Transcription Factors
- Transplant-Ready? Evaluating AI Lung Segmentation Models in Candidates with Severe Lung Disease
- A Trans-Diagnostic Investigation of Attention and Diverse Phenotypes of “Auditory Hyperreactivity” in Autism, ADHD, and the General Population
- Calibrating MLLM-as-a-judge via Multimodal Bayesian Prompt Ensembles
- Bringing Closure to False Discovery Rate Control: A General Principle for Multiple Testing
- RISE: Two‐Stage Rank‐Based Identification of High‐Dimensional Surrogate Markers Applied to Vaccinology
- Diverse feasting networks at the end of the Bronze Age in Britain (c. 900-500 BCE) evidenced by multi-isotope analysis
- EffortNet: A Deep Learning Framework for Objective Assessment of Speech Enhancement Technologies Using EEG-Based Alpha Oscillations
- Principled Detection of Hallucinations in Large Language Models via Multiple Testing
- ELATE: Evolutionary Language model for Automated Time-series Engineering
- Online Conformal Selection with Accept-to-Reject Changes
- Beware of counter-intuitive levels of false discoveries in datasets with strong intra-correlations
- Bridging Human and LLM Judgments: Understanding and Narrowing the Gap
- Revealing the brain signal complexity underlying superior putting performance in expert golfers: A multiscale entropy study with supplemental connectivity analyses
- Enhanced climate reproducibility testing with false discovery rate correction
- Conservation in the Anthropocene: Using Genetics to Understand the Past and Manage for the Future of the Threatened Stephens' Kangaroo Rat
- False discovery rate control with compound p-values
- The effective connectome over a century of human life
- Conformal Data Contamination Tests for Trading or Sharing of Data
- GASPnet: Global Agreement to Synchronize Phases
- Genomic Insights Into Population Structure and Demographic History of White Anglerfish ( Lophius piscatorius ) Throughout Its Range
- Class conditional conformal prediction for multiple inputs by p-value aggregation
- Persistent Convolution: A Topological Framework for AI Alignment Testing and Semantic Space Characterization
- In silico genomic surveillance by CoVerage predicts and characterizes SARS-CoV-2 variants of interest
- Representation learning with a transformer by contrastive learning for money laundering detection
- Large-scale composite hypothesis testing procedure for omics data analyses
- Gold after Randomized Sand: Model-X Split Knockoffs for Controlled Transformation Selection
- New Tests of Randomness for Circular Data
- Multiple Hypothesis Testing in Genomics
- Evaluating Randomness Assumption: A Novel Graph Theoretic Approach
- Specialized response of default mode subnetworks and multiple-demand regions to changes in social content, place and time
- Change-Points Detection and Support Recovery for Spatially Indexed Functional Data
- The Effects of GitHub Copilot on Computing Students' Programming Effectiveness, Efficiency, and Processes in Brownfield Programming Tasks
- SAFER: A Calibrated Risk-Aware Multimodal Recommendation Model for Dynamic Treatment Regimes
- An atlas of exposome-phenome associations in health and disease risk
- Joint mean and covariance estimation with unreplicated matrix-variate data
- A New Procedure for Controlling False Discovery Rate in Large-Scale t-tests
- Explosion of formulaic research articles, including inappropriate study designs and false discoveries, based on the NHANES US national health database
- The phenomenological characteristics of autobiographical future thinking in dysphoric and non-dysphoric individuals
- Interactive Learning of Environment Dynamics for Sequential Tasks
- Directional FDR Control for Sub-Gaussian Sparse GLMs
- Analysis of Cortical Morphometric Variability Using Labeled Cortical Distance Maps
- Robust domain selection for functional data via interval-wise testing and effect size mapping
- FN400 and sustained negativity reveal a premise monotonicity effect during semantic category-based induction
- Computationally Efficient Feature Significance and Importance for Machine Learning Models
- e-GAI: e-value-based Generalized α-Investing for Online False Discovery Rate Control
- A Bandit Approach to Multiple Testing with False Discovery Control
- Consistent Variable Selection for Functional Regression Models
- Changes in Evolutionary Developmental Control Points in the Amniote Limb May Explain Hyperphalangy
- Textual Analysis in Real Estate
- An Efficient Rigorous Approach for Identifying Statistically Significant Frequent Itemsets
- Many Phenotypes without Many False Discoveries: Error Controlling Strategies for Multi-Traits Association Studies
- Feature selection for high-dimensional integrated data
- Inference for Individual Mediation Effects and Interventional Effects in Sparse High-Dimensional Causal Graphical Models
- Ageing and reproduction: antioxidant supplementation alleviates telomere loss in wild birds
- Stable Feature Selection with Applications to MALDI Imaging Mass\n Spectrometry Data
- Post-Model-Selection Statistical Inference with Interrupted Time Series Designs: An Evaluation of an Assault Weapons Ban in California
- False Discovery Rate Computation: Illustrations and Modifications
- A more practical approach for the Benjamini-Hochberg FDR controlling procedure for huge-scale testing problems
- Smaller p-values via indirect information
- Semantic Alignment of AI Models: Concept Collapse, Checkpoint Dynamics, and Cross-Lingual Transfer
- Population‐level gene expression can repeatedly link genes to functions in maize
- Anytime-valid simultaneous lower confidence bounds for the true discovery proportion
- A sufficient criterion for control of generalised error rates in multiple testing
- Decoding the physics of observed actions in the human brain
- Controlling the false discovery rate in high-dimensional linear models using model-X knockoffs and p-values
- Efficient statistical analysis of large correlated multivariate datasets: a case study on brain connectivity matrices
- Sharp Asymptotic Minimaxity for Multiple Testing Using One-Group Shrinkage Priors
- ARM: Detector-Agnostic Changepoint Attribution with Finite-Sample Error Control
- Longitudinal Support Vector Machines for High Dimensional Time Series
- Finding Distributions that Differ, with False Discovery Rate Control
- Extensive genome heterogeneity leads to preferential allele expression and copy number‐dependent expression in cultivated potato
- PsyMem: Fine-grained psychological alignment and Explicit Memory Control for Advanced Role-Playing LLMs
- scikit-covtest: Covariance Matrix Hypothesis Testing in Python
- Compound p-Value Statistics for Multiple Testing Procedures
- Gaussian Graphical Model Estimation with False Discovery Rate Control
- A New Step-down Procedure for Simultaneous Hypothesis Testing Under Dependence
- Multiple Inference and Gender Differences in the Effects of Early Intervention: A Reevaluation of the Abecedarian, Perry Preschool, and Early Training Projects
- Sexual Selection and Predator Response in a Male‐Polymorphic Livebearing Fish
- Effects of coconut oil, olive oil, and butter on plasma fatty acids and metabolic risk factors: a randomized trial
- Modifications of the Helmintex Method: Integration of Treatments With Tween 20, Cellulase, Percoll and Auramine‐O Staining for Detection of Schistosoma mansoni Eggs
- Terrorist Attacks and Trust in Institutions: Micro Evidence From Europe
- Population genetics of Glossina palpalis gambiensis in the sleeping sickness focus of Boffa (Guinea) before and after eight years of vector control: no effect of control despite a significant decrease of human exposure to the disease
- SubROC: AUC-Based Discovery of Exceptional Subgroup Performance for Binary Classifiers
- An Optimal False Discovery Rate Controlling Procedure for Changepoint Detection
- Drivers of morphological evolution in the toothed whale jaw
- Selectivity for food in human ventral visual cortex
- How much does your data exploration overfit? Controlling bias via information usage
- Two-directional simultaneous inference for high-dimensional models
- The effects of tree diversity and neighborhood on phyllosphere fungal communities
- Microsatellite loci support recent speciation of Ambylomma maculatum s.l., Koch, 1944 (Acari: Ixodidae) morphotypes II and III
- Introduction to Random Fields
- The Significant Effects of Threshold Selection for Advancing Nitrogen Use Efficiency in Whole Genome of Bread Wheat
- Context-Aware Online Conformal Anomaly Detection with Prediction-Powered Data Acquisition
- Strong Control, Conservative Point Estimation and Simultaneous Conservative Consistency of False Discovery Rates: A Unified Approach
- Bayesian Model for Multiple Change-points Detection in Multivariate Time Series
- A stochastic process approach to false discovery control
- False discovery rate control with multivariate p-values
- Control of generalized error rates in multiple testing
- Testing for replicability in a follow-up study when the primary study hypotheses are two-sided
- Demographically-Based Evaluation of Genomic Regions under Selection in Domestic Dogs
- A statistical methodology to select covariates in high-dimensional data under dependence. Application to the classification of genetic profiles in oncology
- Statistical Inference After Model Selection
- A News-based Machine Learning Model for Adaptive Asset Pricing
- On false discovery control under dependence
- Generalizing Simes’ test and Hochberg’s stepup procedure
- Geography is not destiny: A quantitative test of Diamond's axis of orientation hypothesis
- Multiple Testing and Error Control in Gaussian Graphical Model Selection
- Microarrays, Empirical Bayes and the Two-Groups Model
- KnowEEG: Explainable Knowledge Driven EEG Classification
- Interactions between nascent proteins translated by adjacent ribosomes drive homomer assembly
- NApy: Efficient Statistics in Python for Large-Scale Heterogeneous Data with Enhanced Support for Missing Data
- Interhemispheric transfer of working memories
- The testing effect in free recall is associated with enhanced organizational processes
- Differential Treatment of Female Defendants
- Measuring impacts of conservation interventions on human well-being and the environment in Northern Cambodia
- Asynchronous Online Testing of Multiple Hypotheses
- Wide-Scale Analysis of Human Functional Transcription Factor Binding Reveals a Strong Bias towards the Transcription Start Site
- Functional Modularity of Background Activities in Normal and Epileptic Brain Networks
- Statistical estimation and testing via the sorted L1 norm
- Comment: Microarrays, Empirical Bayes and the Two-Groups Model
- Comment: Microarrays, Empirical Bayes and the Two-Group Model
- Asymptotic properties of false discovery rate controlling procedures under independence
- Decision theory results for one-sided multiple comparison procedures
- Characterization of Bayes procedures for multiple endpoint problems and inadmissibility of the step-up procedure
- Fast approximate loss change for active learning on imbalanced data
- Integrating learning platforms within regular school time: experimental evidence from Chilean primary schools
- All roads lead to home: panmixia of European eel in the Sargasso Sea
- The evolutionary rate of antibacterial drug targets
- A genome-wide analysis of the shared genetic risk architecture of complex neurological and psychiatric disorders
- Synthetic-Powered Multiple Testing with FDR Control
- INVITED REVIEW: What is a population? An empirical evaluation of some genetic methods for identifying the number of gene pools and their degree of connectivity
- Noisy splicing, more than expression regulation, explains why some exons are subject to nonsense-mediated mRNA decay
- Comprehensive assessment of side effects associated with a single dose of ketamine in treatment-resistant depression
- HERMES: Towards an Integrated Toolbox to Characterize Functional and Effective Brain Connectivity
- Metabolomics: A Global Biochemical Approach to Drug Response and Disease
- The Benjamini--Hochberg Procedure Can Fail to Control the FDR for Correlated Two-Sided Gaussian Tests
- The tempo of cetacean cranial evolution
- Confabulation in schizophrenia: A neuropsychological study
- Predictors of phenotypic progression and disease onset in premanifest and early-stage Huntington's disease in the TRACK-HD study: analysis of 36-month observational data
- Tick-Bacteria Mutualism Depends on B Vitamin Synthesis Pathways
- The Dawn of Human Matrilineal Diversity
- Population structure over a broad spatial scale driven by nonanthropogenic factors in a wide‐ranging migratory mammal, Alaskan caribou
- A comparative study of craniofacial measurements between Ryukyuan and mainland Japanese females using lateral cephalometric images
- Differentially Private False Discovery Rate Control
- Morphological variation in Homo erectus and the origins of developmental plasticity
- Safe Urban Traffic Control via Uncertainty-Aware Conformal Prediction and World-Model Reinforcement Learning
- Type I error rate control for testing many hypotheses: a survey with proofs
- Shared proteomic effects of cerebral atherosclerosis and Alzheimer’s disease on the human brain
- Leveraging green infrastructure for efficient treatment of reclaimed water
- The positive false discovery rate: a Bayesian interpretation and the q-value
- On the false discovery rate and an asymptotically optimal rejection curve
- The nature of social dominance orientation: Theorizing and measuring preferences for intergroup inequality using the new SDO₇ scale.
- Genome-wide mapping of global-to-local genetic effects on human facial shape
- Accounting for ancestry: population substructure and genome-wide association studies
- A Review on Algorithms for Constraint-based Causal Discovery
- Global and Simultaneous Hypothesis Testing for High-Dimensional Logistic Regression Models
- False Discovery Rate Controlled Heterogeneous Treatment Effect Detection for Online Controlled Experiments
- BCI learning induces core-periphery reorganization in M/EEG multiplex brain networks
- Mother's education and child development: Evidence from the compulsory school reform in China
- Genome-Wide Significance Levels and Weighted Hypothesis Testing
- Skilled Mutual Fund Selection: False Discovery Control under Dependence
- Dependency and false discovery rate: Asymptotics
- A Significance Test for Graph-Constrained Estimation
- Simulation-Based Hypothesis Testing of High Dimensional Means under Covariance Heterogeneity
- Measuring the Three‐Point Correlation Function of the Cosmic Microwave Background
- The e-Partitioning Principle of False Discovery Rate Control
- A comparison of machine learning algorithms for the surveillance of autism spectrum disorder
- Goodness-of-fit tests for the functional linear model based on randomly projected empirical processes
- Hierarchy of Neural Organization in the Embryonic Spinal Cord: Granger-Causality Graph Analysis of In Vivo Calcium Imaging Data
- Multivariate tests of association based on univariate tests
- False discovery rate control for multiple testing based on discretep‐values
- Uniformly consistently estimating the proportion of false null hypotheses via Lebesgue–Stieltjes integral equations
- Online Control of the False Discovery Rate under "Decision Deadlines"
- A Geometrical–Statistical Approach to Outlier Removal for TDOA Measurements
- FDR-control in multiscale change-point segmentation
- Estimation of large covariance and precision matrices from temporally dependent observations
- Spatial Confidence Regions for Excursion Sets with False Discovery Rate Control
- Semiparametric Causal Discovery and Inference with Invalid Instruments
- Bringing closure to FDR control: beating the e-Benjamini-Hochberg procedure
- High-dimensional mediation analysis for survival data
- SCORE: A Unified Framework for Overshoot Refund in Online FDR Control
- Stepup procedures controlling generalized FWER and generalized FDR
- LIFE 4 Pollinators’ platform: How citizen science can help monitoring plants and pollinators
- Feature aware covariance estimation, with application to mixtures of chemical exposures
- High-precision binary trait association on phylogenetic trees
- Sequential Change Point Detection with FDR Control in Reconfigurable Sensor Networks
- Three-dimensional cardiovascular imaging-genetics: a mass univariate framework