Class Similarity-Based Multimodal Classification under Heterogeneous Category Sets

2025/06/11 by Zhu, Yangrui, Bao, Junhua, Yapeng Li +4
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Text and Document Classification Technologies

paper · pdf · doi:10.48550/arxiv.2506.09745

openalex publication_date 2025/06/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Existing multimodal methods typically assume that different modalities share the same category set. However, in real-world applications, the category distributions in multimodal data exhibit inconsistencies, which can hinder the model's ability to effectively utilize cross-modal information for recognizing all categories. In this work, we propose the practical setting termed Multi-Modal Heterogeneous Category-set Learning (MMHCL), where models are trained in heterogeneous category sets of multi-modal data and aim to recognize complete classes set of all modalities during test. To effectively address this task, we propose a Class Similarity-based Cross-modal Fusion model (CSCF). Specifically, CSCF aligns modality-specific features to a shared semantic space to enable knowledge transfer between seen and unseen classes. It then selects the most discriminative modality for decision fusion through uncertainty estimation. Finally, it integrates cross-modal information based on class similarity, where the auxiliary modality refines the prediction of the dominant one. Experimental results show that our method significantly outperforms existing state-of-the-art (SOTA) approaches on multiple benchmark datasets, effectively addressing the MMHCL task.

Citations

• Trusted Unified Feature-Neighborhood Dynamics for Multi-View Classification
• Audio-Visual Generalized Zero-Shot Learning using Pre-Trained Large Multi-Modal Models
• Hyperbolic Audio-visual Zero-shot Learning
• Zero-Shot Learning by Harnessing Adversarial Samples
• Evolving Semantic Prototype Improves Generative Zero-Shot Learning
• GPT-4 Technical Report
• Incomplete Multi-View Multi-Label Learning via Label-Guided Masked View- and Category-Aware Transformers
• Trusted Multi-View Classification with Dynamic Evidential Fusion
• Zero-Shot Logit Adjustment
• Deconstructed Generation-Based Zero-Shot Model
• Audio-visual Generalised Zero-shot Learning with Cross-modal Attention and Language
• MSDN: Mutually Semantic Distillation Network for Zero-Shot Learning
• FREE: Feature Refinement for Generalized Zero-Shot Learning
• What Makes Multi-modal Learning Better than Single (Provably)
• UniT: Multimodal Multitask Learning with a Unified Transformer
• A Variational Information Bottleneck Approach to Multi-Omics Data Integration
• Deep Multimodal Fusion by Channel Exchanging
• More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classification
• Survey on Deep Multi-modal Data Analytics: Collaboration, Rivalry and Fusion
• AVGZSLNet: Audio-Visual Generalized Zero-Shot Learning by Reconstructing Label Features from Multi-Modal Embeddings
• Greedy Policy Search: A Simple Baseline for Learnable Test-Time Augmentation
• Supervised Multimodal Bitransformers for Classifying Images and Text
• Multimodal Transformer for Unaligned Multimodal Language Sequences
• Multimodal Classification of Urban Micro-Events
• Semantically Aligned Bias Reducing Zero Shot Learning
• Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders
• Evidential Deep Learning to Quantify Classification Uncertainty
• Feature Generating Networks for Zero-Shot Learning
• Deep Multimodal Learning: A Survey on Recent Advances and Trends
• VIGAN: Missing View Imputation with Generative Adversarial Networks
• Multimodal Machine Learning: A Survey and Taxonomy
• Multimodal Machine Learning: A Survey and Taxonomy
• Zero-Shot Recognition using Dual Visual-Semantic Mapping Paths
• Gated Multimodal Units for Information Fusion
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Majority Vote of Diverse Classifiers for Late Fusion

Related

• Joint Representation Learning and Novel Category Discovery on Single- and Multi-modal Data
• Generative Modeling of Class Probability for Multi-Modal Representation Learning
• Open-set Cross Modal Generalization via Multimodal Unified Representation
• Rethinking Multimodal Learning from the Perspective of Mitigating Classification Ability Disproportion
• Learning Discriminative Representations for Semantic Cross Media Retrieval