2025/08/01 by Ben Chugg, Tyron Lardy, Chugg, Ben +5 · 3 citations
Mathematics · Decision Sciences · Computer Science · #Statistical Methods in Clinical Trials #Advanced Statistical Process Monitoring #Distributed Sensor Networks and Detection Algorithms
paper · pdf · doi:10.48550/arxiv.2508.00770
The validity of classical hypothesis testing requires the significance level α be fixed before any statistical analysis takes place. This is a stringent requirement. For instance, it prohibits updating α during (or after) an experiment due to changing concern about the cost of false positives, or to reflect unexpectedly strong evidence against the null. Perhaps most disturbingly, witnessing a p-value p≪α vs p= α- ε for tiny ε> 0 has no (statistical) relevance for any downstream decision-making. Following recent work of Grünwald (2024), we develop a theory of post-hoc hypothesis testing, enabling α to be chosen after seeing and analyzing the data. To study "good" post-hoc tests we introduce Γ-admissibility, where Γ is a set of adversaries which map the data to a significance level. We classify the set of Γ-admissible rules for various sets Γ, showing they must be based on e-values, and recover the Neyman-Pearson lemma when Γ is the constant map.