CLARGA: Multimodal Graph Representation Learning over Arbitrary Sets of Modalities

2025/12/10 by Santosh Patapati, Patapati, Santosh
Computer Science · Psychology · #Advanced Graph Neural Networks #Computer Vision and Pattern Recognition (cs.CV) #Emotion and Mood Recognition #FOS: Computer and information sciences #Machine Learning (cs.LG) #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2512.11901

openalex publication_date 2025/12/10 · openalex created_date 2025/12/17 · openalex updated_date 2026/07/28

Abstract

We introduce CLARGA, a general-purpose multimodal fusion architecture for multimodal representation learning that works with any number and type of modalities without changing the underlying framework. Given a supervised dataset, CLARGA can be applied to virtually any machine learning task to fuse different multimodal representations for processing by downstream layers. On a sample-by-sample basis, CLARGA learns how modalities should inform one another by building an attention weighted graph over their features and passing messages along this graph with a multi-head Graph Attention Network. Not only does this make CLARGA highly adaptive, as it constructs unique graphs for different samples, it makes for efficient fusion with sub-quadratic complexity as the number of modalities grows. Through a learnable mask, it can also adapt to missing modality inputs. The model is trained with a hybrid objective that combines a supervised task loss with contrastive InfoNCE loss, improving cross-modal consistency and robustness to noisy inputs. We demonstrate CLARGA's effectiveness in diverse multimodal representation learning tasks across 7 datasets spanning finance, human-computer interaction, general multimedia classification, and affective computing. It consistently outperforms baselines, state-of-the-art models, and ablations. Additional experiments also demonstrate its robustness to missing inputs and ability to excel on niche tasks. Overall, CLARGA can be easily plugged into machine learning models for effective and efficient learning of representations across a wide variety of tasks.

Citations

• CLIP-MG: Guiding Semantic Attention with Skeletal Pose Features and RGB Data for Micro-Gesture Recognition on the iMiGUE Dataset
• Multimodal Federated Learning With Missing Modalities through Feature Imputation Network
• Multimodal Fusion with Relational Learning for Molecular Property Prediction
• Multimodal Representation Learning using Adaptive Graph Construction
• MMP: Towards Robust Multi-Modal Learning with Masked Modality Projection
• ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Area
• Residual Connections and Normalization Can Provably Prevent Oversmoothing in GNNs
• Multimodal Lego: Model Merging and Fine-Tuning Across Topologies and Modalities in Biomedicine
• CICA: Content-Injected Contrastive Alignment for Zero-Shot Document Image Classification
• Exploring Missing Modality in Multimodal Egocentric Datasets
• Data-Efficient Multimodal Fusion on a Single GPU
• AV-SUPERB: A Multi-Task Evaluation Benchmark for Audio-Visual Representation Models
• One-Versus-Others Attention: Scalable Multimodal Integration for Biomedical Data
• Missing Modality Robustness in Semi-Supervised Multi-Modal Semantic Segmentation
• Towards Good Practices for Missing Modality Robust Action Recognition
• Are Multimodal Transformers Robust to Missing Modality?
• A Comparative Analysis of Decision-Level Fusion for Multimodal Driver Behaviour Understanding
• Robust Graph Neural Networks via Probabilistic Lipschitz Constraints
• MultiBench: Multiscale Benchmarks for Multimodal Representation Learning
• Attention Bottlenecks for Multimodal Fusion
• Deep Learning for Depression Recognition with Audiovisual Cues: A Review
• Learning Transferable Visual Models From Natural Language Supervision
• ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks
• Multimodal Transformer for Unaligned Multimodal Language Sequences
• MFAS: Multimodal Fusion Architecture Search
• On the Universality of Invariant Networks
• Learning deep representations by mutual information estimation and maximization
• Representation Learning with Contrastive Predictive Coding
• Learning Factorized Multimodal Representations
• Deep Bidirectional and Unidirectional LSTM Recurrent Neural Network for Network-wide Traffic Speed Prediction
• Graph Attention Networks
• Multimodal Machine Learning: A Survey and Taxonomy
• Multimodal Machine Learning: A Survey and Taxonomy
• Deep Sets
• Deep Generalized Canonical Correlation Analysis
• Gated Multimodal Units for Information Fusion
• Attention-Based Multimodal Fusion for Video Description
• AVEC 2016 - Depression, Mood, and Emotion Recognition Workshop and Challenge
• Deep Residual Learning for Image Recognition
• Very Deep Convolutional Networks for Large-Scale Image Recognition
• Efficient Estimation of Word Representations in Vector Space
• High-Performance Neural Networks for Visual Object Classification
• The PHQ-8 as a measure of current depression in the general population
• Multilayer feedforward networks are universal approximators
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
• Gradient-based learning applied to document recognition
• A Scale for the Measurement of the Psychological Magnitude Pitch

Related

• Multimodal Representation Learning using Adaptive Graph Construction
• MMGA: Multimodal Learning with Graph Alignment
• Multimodal Representation Learning and Fusion
• Graph4MM: Weaving Multimodal Learning with Structural Information
• Graph-MLLM: Harnessing Multimodal Large Language Models for Multimodal Graph Learning