Decoupling Common and Unique Representations for Multimodal Self-supervised Learning

2023/09/11 by Wang, Yi, Albrecht, Conrad M, Braham, Nassim Ait Ali +3 · 10 citations
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2309.05300

Abstract

The increasing availability of multi-sensor data sparks wide interest in multimodal self-supervised learning. However, most existing approaches learn only common representations across modalities while ignoring intra-modal training and modality-unique representations. We propose Decoupling Common and Unique Representations (DeCUR), a simple yet effective method for multimodal self-supervised learning. By distinguishing inter- and intra-modal embeddings through multimodal redundancy reduction, DeCUR can integrate complementary information across different modalities. We evaluate DeCUR in three common multimodal scenarios (radar-optical, RGB-elevation, and RGB-depth), and demonstrate its consistent improvement regardless of architectures and for both multimodal and modality-missing settings. With thorough experiments and comprehensive analysis, we hope this work can provide valuable insights and raise more interest in researching the hidden relationships of multimodal representations.

Cited by

• Structured Spectral Reasoning for Frequency-Adaptive Multimodal Recommendation
• GEO-Bench-2: From Performance to Capability, Rethinking Evaluation in Geospatial AI
• LandSegmenter: Towards a Flexible Foundation Model for Land Use and Land Cover Mapping
• GeoCrossBench: Cross-Band Generalization for Remote Sensing
• Assessing the value of Geo-Foundational Models for Flood Inundation Mapping: Benchmarking models for Sentinel-1, Sentinel-2, and Planetscope for end-users
• Multi-modal Co-learning for Earth Observation: Enhancing single-modality models via modality collaboration
• GeoLink: Empowering Remote Sensing Foundation Model with OpenStreetMap Data
• Large scale cross-regional remote sensing flood monitoring framework for operative mapping and impact analysis
• FAIRWELL: Fair Multimodal Self-Supervised Learning for Wellbeing Prediction
• MAESTRO: Masked AutoEncoders for Multimodal, Multitemporal, and Multispectral Earth Observation Data

Related

• Cross and Learn: Cross-Modal Self-Supervision
• What to align in multimodal contrastive learning?
• Self-Supervised Multimodal Learning: A Survey
• MultiLoReFT: Decoupling Shared and Modality-Specific Subspaces in Multimodal Learning via Low-Rank Representation Fine-Tuning
• A survey on Self Supervised learning approaches for improving Multimodal representation learning