Toward Deconfounding the Influence of Entity Demographics for Question Answering Accuracy

2021/04/15 by Maharshi Gor, Kellie Webster, Gor, Maharshi +3 · 1 citation
Computer Science · #Topic Modeling #Expert finding and Q&A systems #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2104.07571

Abstract

The goal of question answering (QA) is to answer any question. However, major QA datasets have skewed distributions over gender, profession, and nationality. Despite that skew, model accuracy analysis reveals little evidence that accuracy is lower for people based on gender or nationality; instead, there is more variation on professions (question topic). But QA's lack of representation could itself hide evidence of bias, necessitating QA datasets that better represent global diversity.

Citations

• A BERT Baseline for the Natural Questions

Cited by

• Open-DeBias: Toward Mitigating Open-Set Bias in Language Models

Related

• Emerging Challenges in Personalized Medicine: Assessing Demographic Effects on Biomedical Question Answering Systems
• BBQ: A Hand-Built Bias Benchmark for Question Answering
• UnQovering Stereotyping Biases via Underspecified Questions
• Social Bias in Popular Question-Answering Benchmarks
• DAIQ: Auditing Demographic Attribute Inference from Question in LLMs