AetherVision-Bench: An Open-Vocabulary RGB-Infrared Benchmark for Multi-Angle Segmentation across Aerial and Ground Perspectives

2025/06/04 by Aniruddh Sikdar, Sikdar, Aniruddh, Akash Gandhamal +3
Computer Science · #Advanced Neural Network Applications #Benchmark (surveying) #Domain Adaptation and Few-Shot Learning #Drone #Image segmentation #Key (lock) #Multimodal Machine Learning Applications #Pixel #Robustness (evolution) #Segmentation

paper · pdf · doi:10.48550/arxiv.2506.03709

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/06/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Open-vocabulary semantic segmentation (OVSS) involves assigning labels to each pixel in an image based on textual descriptions, leveraging world models like CLIP. However, they encounter significant challenges in cross-domain generalization, hindering their practical efficacy in real-world applications. Embodied AI systems are transforming autonomous navigation for ground vehicles and drones by enhancing their perception abilities, and in this study, we present AetherVision-Bench, a benchmark for multi-angle segmentation across aerial, and ground perspectives, which facilitates an extensive evaluation of performance across different viewing angles and sensor modalities. We assess state-of-the-art OVSS models on the proposed benchmark and investigate the key factors that impact the performance of zero-shot transfer models. Our work pioneers the creation of a robustness benchmark, offering valuable insights and establishing a foundation for future research.

Citations

• SAGA: Semantic-Aware Gray color Augmentation for Visible-to-Thermal Domain Adaptation across Multi-View Drone and Ground-Based Vision Systems
• A Resource-Efficient Decentralized Sequential Planner for Spatiotemporal Wildfire Mitigation
• LCA-on-the-Line: Benchmarking Out-of-Distribution Generalization with Class Taxonomies
• Caltech Aerial RGB-Thermal Dataset in the Wild
• MRFP: Learning Generalizable Semantic Segmentation from Sim-2-Real with Multi-Resolution Feature Perturbation
• DePT: Decoupled Prompt Tuning
• Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP
• What a MESS: Multi-Domain Evaluation of Zero-Shot Semantic Segmentation
• CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation
• Side Adapter Network for Open-Vocabulary Semantic Segmentation
• Generalized Decoding for Pixel, Image, and Language
• Fully Complex-valued Fully Convolutional Multi-feature Fusion Network (FC2MFN) for Building Segmentation of InSAR images
• An Efficient Approach with Dynamic Multi-Swarm of UAVs for Forest Firefighting
• Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP
• Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution
• A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-language Model
• Scaling Open-Vocabulary Image Segmentation with Image-Level Labels
• Extract Free Dense Labels from CLIP
• Florence: A New Foundation Model for Computer Vision
• Robust fine-tuning of zero-shot models
• Learning Transferable Visual Models From Natural Language Supervision
• UAVid: A semantic segmentation dataset for UAV imagery
• COCO-Stuff: Thing and Stuff Classes in Context
• Semantic Understanding of Scenes through the ADE20K Dataset
• Semantic Understanding of Scenes Through the ADE20K Dataset

Related

• RIS-LAD: A Benchmark and Model for Referring Low-Altitude Drone Image Segmentation
• PPTFormer: Pseudo Multi-Perspective Transformer for UAV Segmentation
• PixDLM: A Dual-Path Multimodal Language Model for UAV Reasoning Segmentation
• High-Resolution Multispectral Dataset for Semantic Segmentation
• IndraEye: Infrared Electro-Optical UAV-based Perception Dataset for Robust Downstream Tasks