Multimodal Learning with Augmentation Techniques for Natural Disaster Assessment

2025/10/04 by Adrian-Dinu Urse, Urse, Adrian-Dinu, Dumitru-Clementin Cercel +3
Computer Science · Social Sciences · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #Computers and Society (cs.CY) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Public Relations and Crisis Communication #Seismology and Earthquake Studies

paper · pdf · doi:10.48550/arxiv.2511.00004

openalex publication_date 2025/10/04 · openalex created_date 2025/11/05 · openalex updated_date 2026/07/28

Abstract

Natural disaster assessment relies on accurate and rapid access to information, with social media emerging as a valuable real-time source. However, existing datasets suffer from class imbalance and limited samples, making effective model development a challenging task. This paper explores augmentation techniques to address these issues on the CrisisMMD multimodal dataset. For visual data, we apply diffusion-based methods, namely Real Guidance and DiffuseMix. For text data, we explore back-translation, paraphrasing with transformers, and image caption-based augmentation. We evaluated these across unimodal, multimodal, and multi-view learning setups. Results show that selected augmentations improve classification performance, particularly for underrepresented classes, while multi-view learning introduces potential but requires further refinement. This study highlights effective augmentation strategies for building more robust disaster assessment systems.

Citations

• MambaVision: A Hybrid Mamba-Transformer Vision Backbone
• DiffuseMix: Label-Preserving Data Augmentation with Diffusion Models
• CrisisKAN: Knowledge-infused and Explainable Multimodal Attention Network for Crisis Event Classification
• Is synthetic data from generative models ready for image recognition?
• BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
• Incidents1M: a large-scale dataset of images with natural disasters, damage, and incidents
• Deep Learning Benchmarks and Datasets for Social Media Image Classification for Disaster Response
• DeBERTa: Decoding-enhanced BERT with Disentangled Attention
• CrisisBERT: a Robust Transformer for Crisis Classification and Contextual Crisis Embedding
• COBRA: Contrastive Bi-Modal Representation Algorithm
• Analysis of Social Media Data using Multimodal Deep Learning for Disaster Response
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• CrisisMMD: Multimodal Twitter Datasets from Natural Disasters
• Attention Is All You Need
• Densely Connected Convolutional Networks
• Processing Social Media Messages in Mass Emergency: A Survey
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• From Imagery to Insight: Applying Multimodal Large Language Models for Rapid Disaster Impact Assessment
• Differential Attention for Multimodal Crisis Event Analysis
• Analysis of Social Media Data using Multimodal Deep Learning for Disaster Response
• GaLeNet: Multimodal Learning for Disaster Prediction, Management and Relief
• A social context-aware graph-based multimodal attentive learning framework for disaster content classification during emergencies: a benchmark dataset and method