SkySenseGPT: A Fine-Grained Instruction Tuning Dataset and Model for Remote Sensing Vision-Language Understanding

2024/06/14 by Junwei Luo, Zhen Pang, Luo, Junwei +19 · 23 citations
Computer Science · #Advanced Image and Video Retrieval Techniques #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Image Retrieval and Classification Techniques #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2406.10100

openalex publication_date 2024/06/14 · openalex created_date 2024/06/18 · openalex updated_date 2026/07/28

Abstract

Remote Sensing Large Multi-Modal Models (RSLMMs) are developing rapidly and showcase significant capabilities in remote sensing imagery (RSI) comprehension. However, due to the limitations of existing datasets, RSLMMs have shortcomings in understanding the rich semantic relations among objects in complex remote sensing scenes. To unlock RSLMMs' complex comprehension ability, we propose a large-scale instruction tuning dataset FIT-RS, containing 1,800,851 instruction samples. FIT-RS covers common interpretation tasks and innovatively introduces several complex comprehension tasks of escalating difficulty, ranging from relation reasoning to image-level scene graph generation. Based on FIT-RS, we build the FIT-RSFG benchmark. Furthermore, we establish a new benchmark to evaluate the fine-grained relation comprehension capabilities of LMMs, named FIT-RSRC. Based on combined instruction data, we propose SkySenseGPT, which achieves outstanding performance on both public datasets and FIT-RSFG, surpassing existing RSLMMs. We hope the FIT-RS dataset can enhance the relation comprehension capability of RSLMMs and provide a large-scale fine-grained data source for the remote sensing community. The dataset will be available at https://github.com/Luo-Z13/SkySenseGPT

Cited by

• MMLANDMARKS: a Cross-View Instance-Level Benchmark for Geo-Spatial Understanding
• A Benchmark for Ultra-High-Resolution Remote Sensing MLLMs
• Cross-modal Context-aware Learning for Visual Prompt Guided Multimodal Image Understanding in Remote Sensing
• VLM2GeoVec: Toward Universal Multimodal Embeddings for Remote Sensing
• GeoZero: Incentivizing Reasoning from Scratch on Geospatial Scenes
• GeoViS: Geospatially Rewarded Visual Search for Remote Sensing Visual Grounding
• SkyMoE: A Vision-Language Foundation Model for Enhancing Geospatial Interpretation with Mixture of Experts
• UniGeoSeg: Towards Unified Open-World Segmentation for Geospatial Scenes
• Asking like Socrates: Socrates helps VLMs understand remote sensing images
• Co-Training Vision Language Models for Remote Sensing Multi-task Learning
• FedRW: Efficient Privacy-Preserving Data Reweighting for Enhancing Federated Learning of Language Models
• DescribeEarth: Describe Anything for Remote Sensing Images
• GeoVLM-R1: Reinforcement Fine-Tuning for Improved Remote Sensing Reasoning
• Geo-R1: Improving Few-Shot Geospatial Referring Expression Understanding with Reinforcement Fine-Tuning
• Visual Instruction Pretraining for Domain-Specific Foundation Models
• Remote Sensing Image Intelligent Interpretation with the Language-Centered Perspective: Principles, Methods and Challenges
• SPEX: A Vision-Language Model for Land Cover Extraction on Spectral Remote Sensing Images
• Landsat30-AU: A Vision-Language Dataset for Australian Landsat Imagery
• RingMo-Agent: A Unified Remote Sensing Foundation Model for Multi-Platform and Multi-Modal Reasoning
• GeoProg3D: Compositional Visual Reasoning for City-Scale 3D Language Fields
• UrbanLLaVA: A Multi-modal Large Language Model for Urban Intelligence with Spatial Reasoning and Understanding
• Remote Sensing Large Vision-Language Model: Semantic-augmented Multi-level Alignment and Semantic-aware Expert Modeling
• TESSERA: Temporal Embeddings of Surface Spectra for Earth Representation and Analysis

Related

• EarthGPT: A Universal Multi-modal Large Language Model for Multi-sensor Image Comprehension in Remote Sensing Domain
• SkyEyeGPT: Unifying Remote Sensing Vision-Language Tasks via Instruction Tuning with Large Language Model
• Multimodal Large Language Models for Remote Sensing Image Understanding: Domain-Specific or General-Purpose?
• EarthGPT-X: A Spatial MLLM for Multi-level Multi-Source Remote Sensing Imagery Understanding with Visual Prompting
• RSTeller: Scaling Up Visual Language Modeling in Remote Sensing with Rich Linguistic Semantics from Openly Available Data and Large Language Models