2025/03/18 by Meisam Jamshidi Seikavandi, Jostein Fimland, Seikavandi, Meisam Jamshidi +5
Computer Science · Psychology · #Artificial Intelligence (cs.AI) #Emotion and Mood Recognition #FOS: Computer and information sciences #Gaze Tracking and Assistive Technology #Human-Computer Interaction (cs.HC) #Social Robot Interaction and HRI
paper · pdf · doi:10.48550/arxiv.2503.16532
openalex publication_date 2025/03/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Accurate emotion recognition is pivotal for nuanced and engaging human-computer interactions, yet remains difficult to achieve, especially in dynamic, conversation-like settings. In this study, we showcase how integrating eye-tracking data, temporal dynamics, and personality traits can substantially enhance the detection of both perceived and felt emotions. Seventy-three participants viewed short, speech-containing videos from the CREMA-D dataset, while being recorded for eye-tracking signals (pupil size, fixation patterns), Big Five personality assessments, and self-reported emotional states. Our neural network models combined these diverse inputs including stimulus emotion labels for contextual cues and yielded marked performance gains compared to the state-of-the-art. Specifically, perceived valence predictions reached a macro F1-score of 0.76, and models incorporating personality traits and stimulus information demonstrated significant improvements in felt emotion accuracy. These results highlight the benefit of unifying physiological, individual and contextual factors to address the subjectivity and complexity of emotional expression. Beyond validating the role of user-specific data in capturing subtle internal states, our findings inform the design of future affective computing and human-agent systems, paving the way for more adaptive and cross-individual emotional intelligence in real-world interactions.