Retrieval-Augmented Multimodal Depression Detection

2025/10/29 by Hou, Ruibo, Teng, Shiyu, Liu, Jiaqing +4
Computer Science · Psychology · #Computation and Language (cs.CL) #Emotion and Mood Recognition #FOS: Computer and information sciences #Machine Learning (cs.LG) #Mental Health via Writing #Sentiment Analysis and Opinion Mining

paper · doi:10.48550/arxiv.2511.01892

openalex publication_date 2025/10/29 · openalex created_date 2025/11/06 · openalex updated_date 2026/07/28

Abstract

Multimodal deep learning has shown promise in depression detection by integrating text, audio, and video signals. Recent work leverages sentiment analysis to enhance emotional understanding, yet suffers from high computational cost, domain mismatch, and static knowledge limitations. To address these issues, we propose a novel Retrieval-Augmented Generation (RAG) framework. Given a depression-related text, our method retrieves semantically relevant emotional content from a sentiment dataset and uses a Large Language Model (LLM) to generate an Emotion Prompt as an auxiliary modality. This prompt enriches emotional representation and improves interpretability. Experiments on the AVEC 2019 dataset show our approach achieves state-of-the-art performance with CCC of 0.593 and MAE of 3.95, surpassing previous transfer learning and multi-task learning baselines.

Citations

• GPT-4 Technical Report
• Global prevalence and burden of depressive and anxiety disorders in 204 countries and territories in 2020 due to the COVID-19 pandemic
• Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• AVEC 2019 Workshop and Challenge: State-of-Mind, Detecting Depression with AI, and Cross-Cultural Affect Recognition
• Billion-scale similarity search with GPUs
• AVEC 2016 - Depression, Mood, and Emotion Recognition Workshop and Challenge
• A Concordance Correlation Coefficient to Evaluate Reproducibility
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• Exploring Machine Learning and Language Models for Multimodal Depression Detection
• Multimodal Magic Elevating Depression Detection with a Fusion of Text and Audio Intelligence
• Depression Detection and Analysis using Large Language Models on Textual and Audio-Visual Modalities
• Towards Stable Cross-Domain Depression Recognition under Missing Modalities
• Multimodal Domain Generalization for Depression Detection: An Attention-Based BiLSTM Network With Domain-Adversarial Training