Towards Stable Cross-Domain Depression Recognition under Missing Modalities

2025/12/06 by Chen, Jiuyi, Tan, Mingkui, Lu, Haifeng +4
Psychology · #Computer Vision and Pattern Recognition (cs.CV) #Digital Mental Health Interventions #Emotion and Mood Recognition #FOS: Computer and information sciences #Mental Health via Writing

paper · doi:10.48550/arxiv.2512.06447

openalex publication_date 2025/12/06 · openalex created_date 2025/12/10 · openalex updated_date 2026/07/28

Abstract

Depression poses serious public health risks, including suicide, underscoring the urgency of timely and scalable screening. Multimodal automatic depression detection (ADD) offers a promising solution; however, widely studied audio- and video-based ADD methods lack a unified, generalizable framework for diverse depression recognition scenarios and show limited stability to missing modalities, which are common in real-world data. In this work, we propose a unified framework for Stable Cross-Domain Depression Recognition based on Multimodal Large Language Model (SCD-MLLM). The framework supports the integration and processing of heterogeneous depression-related data collected from varied sources while maintaining stability in the presence of incomplete modality inputs. Specifically, SCD-MLLM introduces two key components: (i) Multi-Source Data Input Adapter (MDIA), which employs masking mechanism and task-specific prompts to transform heterogeneous depression-related inputs into uniform token sequences, addressing inconsistency across diverse data sources; (ii) Modality-Aware Adaptive Fusion Module (MAFM), which adaptively integrates audio and visual features via a shared projection mechanism, enhancing resilience under missing modality conditions. e conduct comprehensive experiments under multi-dataset joint training settings on five publicly available and heterogeneous depression datasets from diverse scenarios: CMDC, AVEC2014, DAIC-WOZ, DVlog, and EATD. Across both complete and partial modality settings, SCD-MLLM outperforms state-of-the-art (SOTA) models as well as leading commercial LLMs (Gemini and GPT), demonstrating superior cross-domain generalization, enhanced ability to capture multimodal cues of depression, and strong stability to missing modality cases in real-world applications.

Citations

• MLlm-DR: Towards Explainable Depression Recognition with MultiModal Large Language Models
• AffectGPT: A New Dataset, Model, and Benchmark for Emotion Understanding with Multimodal Large Language Models
• Omni-Emotion: Extending Video MLLM with Detailed Face and Audio Modeling for Multimodal Emotion Analysis
• Facial Expression Analysis and Its Potentials in IoT Systems: A Contemporary Survey
• Understanding Emotional Body Expressions via Large Language Models
• GPT-4o System Card
• DepMamba: Progressive Fusion Mamba for Multimodal Depression Detection
• EmoLLM: Multimodal Emotional Understanding Meets Large Language Models
• Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning
• LMVD: A Large-Scale Multimodal Vlog Dataset for Depression Detection in the Wild
• LLMs are Good Action Recognizers
• Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
• When LLMs Meets Acoustic Landmarks: An Efficient Approach to Integrate Speech into Large Language Models for Depression Detection
• MVBench: A Comprehensive Multi-modal Video Understanding Benchmark
• Automatic Depression Detection: An Emotional Audio-Textual Corpus and a GRU/BiLSTM-based Model
• LoRA: Low-Rank Adaptation of Large Language Models
• Deep Learning for Depression Recognition with Audiovisual Cues: A Review
• NetVLAD: CNN architecture for weakly supervised place recognition

Related

• It Hears, It Sees too: Multi-Modal LLM for Depression Detection By Integrating Visual Understanding into Audio Language Models
• Exploring Machine Learning and Language Models for Multimodal Depression Detection
• Multimodal Domain Generalization for Depression Detection: An Attention-Based BiLSTM Network With Domain-Adversarial Training
• Why Pre-trained Models Fail: Feature Entanglement in Multi-modal Depression Detection
• A Depression Detection Method Based on Multi-Modal Feature Fusion Using Cross-Attention