Dependence Measures Bounding the Exploration Bias for General Measurements

2016/12/18 by Jiantao Jiao, Yanjun Han, Jiao, Jiantao +3 · 1 citation
Computer Science · Mathematics · #FOS: Computer and information sciences #Information Theory (cs.IT) #cs.IT #math.IT

paper · pdf · doi:10.48550/arxiv.1612.05845

arxiv created 2017/07/17 · arxiv updated 2017/07/18

Abstract

We propose a framework to analyze and quantify the bias in adaptive data analysis. It generalizes that proposed by Russo and Zou'15, applying to measurements whose moment generating function exists, measurements with a finite p-norm, and measurements in general Orlicz spaces. We introduce a new class of dependence measures which retain key properties of mutual information while more effectively quantifying the exploration bias for heavy tailed distributions. We provide examples of cases where our bounds are nearly tight in situations where the original framework of Russo and Zou'15 does not apply.

Cited by

• Tail-Aware Information-Theoretic Bounds for LLM Alignment under Heavy-Tailed Rewards

Related

• How much does your data exploration overfit? Controlling bias via information usage
• Learning Dependence Structures for Econometric Inference: Identification, Ambiguity, and Adaptive Inference
• Ranking by Dependence - A Fair Criteria
• Global Sensitivity Analysis with Dependence Measures
• Measuring Dependence between Events