On the Interpretability and Significance of Bias Metrics in Texts: a PMI-based Approach

2021/04/13 by Francisco Valentini, Valentini, Francisco, Germán Rosati +7
Computer Science · #Authorship Attribution and Profiling #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2104.06474

openalex publication_date 2021/04/13 · openalex created_date 2023/07/20 · openalex updated_date 2026/07/28

Abstract

In recent years, word embeddings have been widely used to measure biases in texts. Even if they have proven to be effective in detecting a wide variety of biases, metrics based on word embeddings lack transparency and interpretability. We analyze an alternative PMI-based metric to quantify biases in texts. It can be expressed as a function of conditional probabilities, which provides a simple interpretation in terms of word co-occurrences. We also prove that it can be approximated by an odds ratio, which allows estimating confidence intervals and statistical significance of textual biases. This approach produces similar results to metrics based on word embeddings when capturing gender gaps of the real world embedded in large corpora.

Related

• The Undesirable Dependence on Frequency of Gender Bias Metrics Based on Word Embeddings
• Measuring Societal Biases from Text Corpora with Smoothed First-Order Co-occurrence
• Evaluating Metrics for Bias in Word Embeddings
• Understanding the Origins of Bias in Word Embeddings
• Considerations for the Interpretation of Bias Measures of Word Embeddings