AV-EMO-Reasoning: Benchmarking Emotional Reasoning Capabilities in Omni-modal LLMS with Audio-visual Cues

2025/10/08 by D. Shudong Zhou, Patel, Krish, Krish Patel +28 · 1 citation
Computer Science · Psychology · #Emotion and Mood Recognition #FOS: Computer and information sciences #Multimedia (cs.MM) #Multimodal Machine Learning Applications #Multisensory perception and integration #Sound (cs.SD)

paper · pdf · doi:10.48550/arxiv.2510.07355

openalex publication_date 2025/10/08 · openalex created_date 2025/10/11 · openalex updated_date 2026/07/28

Abstract

Emotions conveyed through voice and face shape engagement and context in human AI interaction. Despite rapid progress in omni modal large language models, the holistic evaluation of emotional reasoning with audiovisual cues remains limited. To address this gap, we introduce AV EMO Reasoning, a benchmark designed to systematically assess emotional reasoning abilities in large language models. The framework uses a curated audiovisual corpus comprising synthetic single turn and multi turn dialogues and a real world subset, together with emotion perception and interaction reasoning metrics, to evaluate whether models can understand user emotions and produce appropriate responses. By releasing a systematic evaluation benchmark, AV EMO Reasoning offers a reproducible standard for evaluating emotion aware dialogue and advances toward more natural, adaptive human AI interaction.

Citations

• EMO-Reasoning: Benchmarking Emotional Reasoning Capabilities in Spoken Dialogue Systems
• LCS-CTC: Leveraging Soft Alignments to Enhance Phonetic Transcription Robustness
• Full-Duplex-Bench v1.5: Evaluating Overlap Handling for Full-Duplex Speech Models
• Towards Accurate Phonetic Error Detection Through Phoneme Similarity Modeling
• K-Function: Joint Pronunciation Transcription and Feedback for Evaluating Kids Language Function
• Seamless Dysfluent Speech Text Alignment for Disordered Speech Analysis
• Analysis and Evaluation of Synthetic Data Generation in Speech Dysfluency Detection
• EmoNet-Face: An Expert-Annotated Benchmark for Synthetic Emotion Recognition
• Dysfluent WFST: A Framework for Zero-Shot Speech Dysfluency Transcription and Detection
• Qwen2.5-Omni Technical Report
• Full-Duplex-Bench: A Benchmark to Evaluate Full-duplex Spoken Dialogue Models on Turn-taking Capabilities
• Baichuan-Omni-1.5 Technical Report
• CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models
• GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot
• Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Tasks
• Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedback
• Freeze-Omni: A Smart and Low Latency Speech-to-speech Dialogue Model with Frozen LLM
• Time and Tokens: Benchmarking End-to-End Speech Dysfluency Detection
• Moshi: a speech-text foundation model for real-time dialogue
• Stimulus Modality Matters: Impact of Perceptual Evaluations from Different Modalities on Speech Emotion Recognition System Performance
• Stutter-Solver: End-to-end Multi-lingual Dysfluency Detection
• LLaMA-Omni: Seamless Speech Interaction with Large Language Models
• SSDM: Scalable Speech Dysfluency Modeling
• Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming
• YOLO-Stutter: End-to-end Region-Wise Speech Dysfluency Detection
• MiniCPM-V: A GPT-4V Level MLLM on Your Phone
• FunAudioLLM: Voice Understanding and Generation Foundation Models for Natural Interaction Between Humans and LLMs
• SD-Eval: A Benchmark Dataset for Spoken Dialogue Understanding Beyond Words
• Can LLMs Understand the Implication of Emphasized Sentences in Dialogue?
• AI-Face: A Million-Scale Demographically Annotated AI-Generated Face Dataset and Fairness Benchmark
• Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversations
• Towards Hierarchical Spoken Language Dysfluency Modeling
• Paralinguistics-Enhanced Large Language Modeling of Spoken Dialogue
• Unconstrained Dysfluency Modeling for Dysfluent Speech Transcription and Detection
• Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speech
• Textually Pretrained Speech Language Models
• GPT-4 Technical Report
• AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representations
• Articulatory Representation Learning Via Joint Factor Analysis and Neural Matrix Factorization
• Deep Neural Convolutive Matrix Factorization for Articulatory Representation Decomposition
• Language Models are Few-Shot Learners
• What comprises a good talking-head video generation?: A Survey and Benchmark
• AVEC 2016 - Depression, Mood, and Emotion Recognition Workshop and Challenge
• DreamTalk: When Emotional Talking Head Generation Meets Diffusion Probabilistic Models

Cited by

• Enabling Conversational Behavior Reasoning Capabilities in Full-Duplex Speech

Related

• EMO-Reasoning: Benchmarking Emotional Reasoning Capabilities in Spoken Dialogue Systems
• Beyond Classification: Towards Speech Emotion Reasoning with Multitask AudioLLMs
• Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning
• EmoLLM: Multimodal Emotional Understanding Meets Large Language Models
• Emotion-Coherent Reasoning for Multimodal LLMs via Emotional Rationale Verifier