Contrastive Language-Image Learning with Augmented Textual Prompts for 3D/4D FER Using Vision-Language Model

2025/04/28 by Muzammil Behzad, Guoying Zhao, Behzad, Muzammil +1 · 1 citation
Computer Science · Psychology · #Convergence (economics) #Emotion and Mood Recognition #Face (sociological concept) #Face recognition and analysis #Feature (linguistics) #Feature learning #Function (biology) #Inference #Multimodal Machine Learning Applications #Representation (politics)

paper · pdf · doi:10.48550/arxiv.2504.19739

openalex publication_date 2025/04/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

In this paper, we introduce AffectVLM, a vision-language model designed to integrate multiviews for a semantically rich and visually comprehensive understanding of facial emotions from 3D/4D data. To effectively capture visual features, we propose a joint representation learning framework paired with a novel gradient-friendly loss function that accelerates model convergence towards optimal feature representation. Additionally, we introduce augmented textual prompts to enhance the model's linguistic capabilities and employ mixed view augmentation to expand the visual dataset. We also develop a Streamlit app for a real-time interactive inference and enable the model for distributed learning. Extensive experiments validate the superior performance of AffectVLM across multiple benchmarks.

Citations

• Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning
• Large Language Models: A Survey
• Vision-Language Models for Vision Tasks: A Survey
• Vision-Language Models for Vision Tasks: A Survey
• AFNet-M: Adaptive Fusion Network with Masks for 2D+3D Facial Expression Recognition
• Learning Transferable Visual Models From Natural Language Supervision
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Towards Reading Beyond Faces for Sparsity-Aware 4D Affect Recognition
• Landmarks-assisted Collaborative Deep Framework for Automatic 4D Facial Expression Recognition
• Magnifying Subtle Facial Motions for Effective 4D Expression Recognition
• Automatic 4D Facial Expression Recognition via Collaborative Cross-domain Dynamic Image Network
• Survey on RGB, 3D, Thermal, and Multimodal Approaches for Facial Expression Recognition: History, Trends, and Affect-related Applications
• Deep Residual Learning for Image Recognition

Cited by

• Facial Emotion Learning with Text-Guided Multiview Fusion via Vision-Language Model for 3D/4D Facial Expression Recognition

Related

• Unsupervised Multiview Contrastive Language-Image Joint Learning with Pseudo-Labeled Prompts Via Vision-Language Model for 3D/4D Facial Expression Recognition
• Self-Supervised Multi-View Representation Learning using Vision-Language Model for 3D/4D Facial Expression Recognition
• Facial Emotion Learning with Text-Guided Multiview Fusion via Vision-Language Model for 3D/4D Facial Expression Recognition
• Training A Small Emotional Vision Language Model for Visual Art Comprehension
• EmoVLM-KD: Fusing Distilled Expertise with Vision-Language Models for Visual Emotion Analysis