SHRUG-FM: Reliability-Aware Foundation Models for Earth Observation

2025/11/13 by Cohrs, Kai-Hendrik, Osika, Zuzanna, Gonzalez-Calabuig, Maria +12
Engineering · Environmental Science · Social Sciences · #Remote-Sensing Image Classification #Remote Sensing in Agriculture #Geographic Information Systems Studies

paper · pdf · doi:10.48550/arxiv.2511.10370

Abstract

Geospatial foundation models (GFMs) for Earth observation often fail to perform reliably in environments underrepresented during pretraining. We introduce SHRUG-FM, a framework for reliability-aware prediction that enables GFMs to identify and abstain from likely failures. Our approach integrates three complementary signals: geophysical out-of-distribution (OOD) detection in the input space, OOD detection in the embedding space, and task-specific predictive uncertainty. We evaluate SHRUG-FM across three high-stakes rapid-mapping tasks: burn scar segmentation, flood mapping, and landslide detection. Our results show that SHRUG-FM consistently reduces prediction risk on retained samples, outperforming established single-signal baselines like predictive entropy. Crucially, by utilizing a shallow "glass-box" decision tree for signal fusion, SHRUG-FM provides interpretable abstention thresholds. It builds a pathway toward safer and more interpretable deployment of GFMs in climate-sensitive applications, bridging the gap between benchmark performance and real-world reliability.

Citations

• Uncertainty evaluation of segmentation models for Earth observation
• AlphaEarth Foundations: An embedding field model for accurate and efficient global mapping from sparse label data
• TESSERA: Temporal Embeddings of Surface Spectra for Earth Representation and Analysis
• Spatially-Aware Evaluation of Segmentation Uncertainty
• REOBench: Benchmarking Robustness of Earth Observation Foundation Models
• ExEBench: Benchmarking Foundation Models on Extreme Earth Events
• TerraMind: Large-Scale Generative Multimodality for Earth Observation
• CARE: Confidence-Aware Regression Estimation of building density fine-tuning EO Foundation Models
• Deep Ensembles Secretly Perform Empirical Bayes
• How Does the Spatial Distribution of Pre-training Data Affect Geospatial Foundation Models?
• Distribution Shifts at Scale: Out-of-distribution Detection in Earth Observation
• PANGAEA: A Global and Inclusive Benchmark for Geospatial Foundation Models
• Prithvi-EO-2.0: A Versatile Multi-Temporal Foundation Model for Earth Observation Applications
• NCDD: Nearest Centroid Distance Deficit for Out-Of-Distribution Detection in Gastrointestinal Vision
• Segmentation Re-thinking Uncertainty Estimation Metrics for Semantic Segmentation
• Analyzing Local Representations of Self-supervised Vision Transformers
• SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery
• An Ambiguity Measure for Recognizing the Unknowns in Deep Learning
• Foundation Models for Generalist Geospatial Artificial Intelligence
• EarthPT: a time series foundation model for Earth Observation
• Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learning
• Decision boundaries and convex hulls in the feature space that deep learning functions learn from images
• Masked Autoencoders Are Scalable Vision Learners
• Machine Learning with a Reject Option: A survey
• A Survey of Uncertainty in Deep Neural Networks
• Emerging Properties in Self-Supervised Vision Transformers
• Momentum Contrast for Unsupervised Visual Representation Learning
• Bias-Reduced Uncertainty Estimation for Deep Neural Classifiers
• On Calibration of Modern Neural Networks
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Deep Neural Networks are Easily Fooled: High Confidence Predictions for\n Unrecognizable Images

Related

• How Does the Spatial Distribution of Pre-training Data Affect Geospatial Foundation Models?
• Uncertainty and Generalizability in Foundation Models for Earth Observation
• GEO-Bench: Toward Foundation Models for Earth Monitoring
• Toward Foundation Models for Earth Monitoring: Proposal for a Climate Change Benchmark
• Foundation Models for Generalist Geospatial Artificial Intelligence