Reading Smiles: Proxy Bias in Foundation Models for Facial Emotion Recognition

2025/06/23 by Tsangko, Iosif, Triantafyllopoulos, Andreas, Abdelmoula, Adem +2
#Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Human-Computer Interaction (cs.HC)

paper · doi:10.48550/arxiv.2506.19079

Abstract

Foundation Models (FMs) are rapidly transforming Affective Computing (AC), with Vision Language Models (VLMs) now capable of recognising emotions in zero shot settings. This paper probes a critical but underexplored question: what visual cues do these models rely on to infer affect, and are these cues psychologically grounded or superficially learnt? We benchmark varying scale VLMs on a teeth annotated subset of AffectNet dataset and find consistent performance shifts depending on the presence of visible teeth. Through structured introspection of, the best-performing model, i.e., GPT-4o, we show that facial attributes like eyebrow position drive much of its affective reasoning, revealing a high degree of internal consistency in its valence-arousal predictions. These patterns highlight the emergent nature of FMs behaviour, but also reveal risks: shortcut learning, bias, and fairness issues especially in sensitive domains like mental health and education.

Citations

• Safety at Scale: A Comprehensive Survey of Large Model and Agent Safety
• Compound Expression Recognition via Large Vision-Language Models
• Emergent Abilities in Large Language Models: A Survey
• Faces of Fairness: Examining Bias in Facial Expression Recognition Datasets and Models
• USER-VLM 360: Personalized Vision Language Models with User-aware Tuning for Social Human-Robot Interactions
• Echo-Teddy: Preliminary Design and Development of Large Language Model-based Social Robot for Autistic Students
• Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling
• Navigating Shortcuts, Spurious Correlations, and Confounders: From Origins via Detection to Mitigation
• Unsupervised Model Diagnosis
• AI-Driven Early Mental Health Screening: Analyzing Selfies of Pregnant Women
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• Affective Computing Has Changed: The Foundation Model Disruption
• Knowledge-Enhanced Facial Expression Recognition with Emotional-to-Neutral Transformation
• Non-Determinism of "Deterministic" LLM Settings
• MiniCPM-V: A GPT-4V Level MLLM on Your Phone
• PaliGemma: A versatile 3B VLM for transfer
• Ovis: Structural Embedding Alignment for Multimodal Large Language Model
• Emergent World Models and Latent Variable Estimation in Chess-Playing Language Models
• InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
• Large Language Models for Robotics: A Survey
• Vision-Language Models for Vision Tasks: A Survey
• Vision-Language Models for Vision Tasks: A Survey
• A Survey of Large Language Models
• Understanding and Mitigating Annotation Bias in Facial Expression Recognition
• On the Opportunities and Risks of Foundation Models
• Generative Adversarial Networks in Human Emotion Synthesis:A Review
• The Clever Hans Effect in Anomaly Detection
• Language Models are Few-Shot Learners
• Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps
• Challenges in Representation Learning: A report on three machine\n learning contests

Related

• Why Do Vision Language Models Struggle To Recognize Human Emotions?
• The Power of Properties: Uncovering the Influential Factors in Emotion Classification
• Beyond Vision: How Large Language Models Interpret Facial Expressions from Valence-Arousal Values
• Balancing the Scales: Enhancing Fairness in Facial Expression Recognition with Latent Alignment
• Faces of Fairness: Examining Bias in Facial Expression Recognition Datasets and Models