Exploring Fusion Techniques in Multimodal AI-Based Recruitment: Insights from FairCVdb

2024/06/17 by Swati, Swati, Roy, Arjun, Ntoutsi, Eirini · 1 citation
#Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Computers and Society (cs.CY) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2407.16892

Abstract

Despite the large body of work on fairness-aware learning for individual modalities like tabular data, images, and text, less work has been done on multimodal data, which fuses various modalities for a comprehensive analysis. In this work, we investigate the fairness and bias implications of multimodal fusion techniques in the context of multimodal AI-based recruitment systems using the FairCVdb dataset. Our results show that early-fusion closely matches the ground truth for both demographics, achieving the lowest MAEs by integrating each modality's unique characteristics. In contrast, late-fusion leads to highly generalized mean scores and higher MAEs. Our findings emphasise the significant potential of early-fusion for accurate and fair applications, even in the presence of demographic biases, compared to late-fusion. Future research could explore alternative fusion strategies and incorporate modality-related fairness constraints to improve fairness. For code and additional insights, visit: https://github.com/Swati17293/Multimodal-AI-Based-Recruitment-FairCVdb

Cited by

• MMM-fair: An Interactive Toolkit for Exploring and Operationalizing Multi-Fairness Trade-offs

Related

• FairCVtest Demo: Understanding Bias in Multimodal Learning with a Testbed in Fair Automatic Recruitment
• Fairness and Bias in Multimodal AI: A Survey
• A Survey on Bias and Fairness in Machine Learning
• Improving Fairness of AI Systems with Lossless De-biasing
• The Multimodal Paradox: How Added and Missing Modalities Shape Bias and Performance in Multimodal AI