Gender Bias in Explainability: Investigating Performance Disparity in Post-hoc Methods

2025/05/02 by Mahdi Dhaini, Dhaini, Mahdi, Ege Erdoğan +5 · 1 citation
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · pdf · doi:10.48550/arxiv.2505.01198

openalex publication_date 2025/05/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

While research on applications and evaluations of explanation methods continues to expand, fairness of the explanation methods concerning disparities in their performance across subgroups remains an often overlooked aspect. In this paper, we address this gap by showing that, across three tasks and five language models, widely used post-hoc feature attribution methods exhibit significant gender disparity with respect to their faithfulness, robustness, and complexity. These disparities persist even when the models are pre-trained or fine-tuned on particularly unbiased datasets, indicating that the disparities we observe are not merely consequences of biased training data. Our results highlight the importance of addressing disparities in explanations when developing and applying explainability methods, as these can lead to biased outcomes against certain subgroups, with particularly critical implications in high-stakes contexts. Furthermore, our findings underscore the importance of incorporating the fairness of explanations, alongside overall model fairness and explainability, as a requirement in regulatory frameworks.

Citations

• GECOBench: A Gender-Controlled Text Dataset and Benchmark for Quantifying Biases in Explanations
• Attention Mechanisms Don't Learn Additive Models: Rethinking Feature Importance for Transformers
• Towards Explainability in Legal Outcome Prediction Models
• Towards Explainability and Fairness in Swiss Judgement Prediction: Benchmarking on a Multilingual Dataset
• Self-AMPLIFY: Improving Small Language Models with Self Post Hoc Explanations
• Understanding Disparities in Post Hoc Machine Learning Explanation
• On the Interplay between Fairness and Explainability
• Bias and Fairness in Large Language Models: A Survey
• Bias and Fairness in Large Language Models: A Survey
• Explainability for Large Language Models: A Survey
• Goodhart's Law Applies to NLP's Explanation Benchmarks
• Post Hoc Explanations of Language Models Can Improve Language Models
• Incorporating Attribution Importance for Improving Faithfulness Metrics
• AtMan: Understanding Transformer Predictions Through Memory Efficient Attention Manipulation
• Trends in Explainable AI (XAI) Literature
• On the Impact of Temporal Concept Drift on Model Explanations
• Towards Faithful Model Explanation in NLP: A Survey
• Challenges in Applying Explainability Methods to Improve the Fairness of NLP Models
• OmniXAI: A Library for Explainable AI
• Perturbation Augmentation for Fairer NLP
• An Empirical Study on Explanations in Out-of-Domain Settings
• The Disagreement Problem in Explainable Machine Learning: A Practitioner's Perspective
• From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AI
• What do we want from Explainable Artificial Intelligence (XAI)? – A stakeholder perspective on XAI and a conceptual model guiding interdisciplinary XAI research
• HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection
• Explainable Automated Fact-Checking: A Survey
• CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models
• Language (Technology) is Power: A Critical Survey of "Bias" in NLP
• Towards Faithfully Interpretable NLP Systems: How should we define and\n evaluate faithfulness?
• On the Legal Compatibility of Fairness Definitions
• "How do I fool you?": Manipulating User Trust via Misleading Black Box Explanations
• ERASER: A Benchmark to Evaluate Rationalized NLP Models
• TinyBERT: Distilling BERT for Natural Language Understanding
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• Attention is not Explanation
• On the (In)fidelity and Sensitivity for Explanations
• Towards Explainable NLP: A Generative Explanation Framework for Text Classification
• On the Robustness of Interpretability Methods
• Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods
• Interpretation of Neural Networks is Fragile
• Towards Deep Learning Models Resistant to Adversarial Attacks
• A Unified Approach to Interpreting Model Predictions
• Axiomatic Attribution for Deep Networks
• "Why Should I Trust You?": Explaining the Predictions of Any Classifier
• Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• Adoption of Explainable Natural Language Processing: Perspectives from Industry and Academia on Practices and Challenges

Related

• Understanding Disparities in Post Hoc Machine Learning Explanation
• GECOBench: A Gender-Controlled Text Dataset and Benchmark for Quantifying Biases in Explanations
• Explanation Bias is a Product: Revealing the Hidden Lexical and Position Preferences in Post-Hoc Feature Attribution
• Gender Encoding Patterns in Pretrained Language Model Representations
• On the Interplay between Fairness and Explainability