The Bias-Expressivity Trade-off

2019/11/09 by Julius Lauw, Dominique Macias, Lauw, Julius +7 · 1 citation
Computer Science · Mathematics · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #cs.AI #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1911.04964

arXiv admin note: text overlap with arXiv:1907.06010

arxiv created 2019/11/09 · arxiv updated 2019/11/13

Abstract

Learning algorithms need bias to generalize and perform better than random guessing. We examine the flexibility (expressivity) of biased algorithms. An expressive algorithm can adapt to changing training data, altering its outcome based on changes in its input. We measure expressivity by using an information-theoretic notion of entropy on algorithm outcome distributions, demonstrating a trade-off between bias and expressivity. To the degree an algorithm is biased is the degree to which it can outperform uniform random sampling, but is also the degree to which is becomes inflexible. We derive bounds relating bias to expressivity, proving the necessary trade-offs inherent in trying to create strongly performing yet flexible algorithms.

Citations

• The Futility of Bias-Free Learning and Search

Cited by

• Model selection in hybrid quantum neural networks with applications to quantum transformer architectures

Related

• The Futility of Bias-Free Learning and Search
• When Are Learning Biases Equivalent? A Unifying Framework for Fairness, Robustness, and Distribution Shift
• A Confidence-Based Approach for Balancing Fairness and Accuracy
• Information-theoretic analysis of generalization capability of learning algorithms
• Machine Learning. The Science of Selection under Uncertainty