GeoFlowVLM: Geometry-Aware Joint Uncertainty for Frozen Vision-Language Embedding

2026/05/31 by Mayank Nautiyal, Li Ju, Andreas Hellander +2
Computer Science · #cs.LG

paper · pdf · doi:10.48550/arxiv.2605.13352

arxiv created 2026/08/01 · arxiv updated 2026/08/04

Abstract

Standard dual-encoder vision-language models that map images and text to deterministic points on a shared unit hypersphere through ℓ₂ normalization typically expose neither aleatoric uncertainty (cross-modal ambiguity) nor epistemic uncertainty (lack of training-distribution support). Existing post-hoc methods either recover at most one of the two uncertainty components, or ignore the hyperspherical geometry of these models' embeddings. We propose GeoFlowVLM as a post-hoc adapter that learns the joint distribution of paired ℓ₂-normalised dual-encoder VLM embeddings on the product hypersphere \mathbbS^d-1 × \mathbbS^d-1 via Riemannian flow matching with a single masked velocity field. A consistency result shows that, in the population limit, the trained network exposes the joint flow and both cross-modal conditional flows as valid Riemannian flow-matching velocity fields on their respective domains. We derive two quantities from this single model: a conditional retrieval entropy that quantifies aleatoric ambiguity with a decision-theoretic interpretation via a Fano-type bound, and a marginal-typicality epistemic score justified by an exact chain-rule decomposition of the joint NLL. This decomposition isolates a cross-modal pointwise-mutual-information term that is structurally discriminative rather than epistemic, and is empirically the only consistently uninformative standalone component. Empirically, the entropy tracks Recall@1 with near-ideal monotonic calibration across three retrieval benchmarks in both directions, and the marginal-typicality sum yields consistently calibrated selective accuracy across four zero-shot classification benchmarks.

Citations

• Exploiting the Asymmetric Uncertainty Structure of Pre-trained VLMs on the Unit Hypersphere
• Probabilistic Embeddings for Frozen Vision-Language Models: Uncertainty Quantification with Gaussian Process Latent Variable Models
• On the Importance of Embedding Norms in Self-Supervised Learning
• Post-hoc Probabilistic Vision-Language Models
• The Double-Ellipsoid Geometry of CLIP
• Probabilistic Language-Image Pre-Training
• Improved Probabilistic Image-Text Representations
• Sigmoid Loss for Language Image Pre-Training
• EVA-CLIP: Improved Training Techniques for CLIP at Scale
• Flow Matching on General Geometries
• Improving and generalizing flow-based generative models with minibatch optimal transport
• Scalable Diffusion Models with Transformers
• Flow Matching for Generative Modeling
• Classifier-Free Diffusion Guidance
• Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning
• Riemannian Score-Based Generative Modelling
• Going deeper with Image Transformers
• Learning Transferable Visual Models From Natural Language Supervision
• Probabilistic Embeddings for Cross-Modal Retrieval
• Riemannian Continuous Normalizing Flows
• Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere
• GLU Variants Improve Transformer
• What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
• Learning Deep Representations of Fine-grained Visual Descriptions
• Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning
• Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models
• Microsoft COCO: Common Objects in Context