2020/02/20 by Duan, Boyan, Ramdas, Aaditya, Wasserman, Larry
#FOS: Computer and information sciences #Methodology (stat.ME)
paper · doi:10.48550/arxiv.2002.08545
We propose a method for multiple hypothesis testing with familywise error rate (FWER) control, called the i-FWER test. Most testing methods are predefined algorithms that do not allow modifications after observing the data. However, in practice, analysts tend to choose a promising algorithm after observing the data; unfortunately, this violates the validity of the conclusion. The i-FWER test allows much flexibility: a human (or a computer program acting on the human's behalf) may adaptively guide the algorithm in a data-dependent manner. We prove that our test controls FWER if the analysts adhere to a particular protocol of "masking" and "unmasking". We demonstrate via numerical experiments the power of our test under structured non-nulls, and then explore new forms of masking.