Hyperbolic Deep Learning for Foundation Models: A Survey

2025/07/23 by He, Neil, Madhu, Hiren, Bui, Ngoc +2 · 5 citations
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2507.17787

Abstract

Foundation models pre-trained on massive datasets, including large language models (LLMs), vision-language models (VLMs), and large multimodal models, have demonstrated remarkable success in diverse downstream tasks. However, recent studies have shown fundamental limitations of these models: (1) limited representational capacity, (2) lower adaptability, and (3) diminishing scalability. These shortcomings raise a critical question: is Euclidean geometry truly the optimal inductive bias for all foundation models, or could incorporating alternative geometric spaces enable models to better align with the intrinsic structure of real-world data and improve reasoning processes? Hyperbolic spaces, a class of non-Euclidean manifolds characterized by exponential volume growth with respect to distance, offer a mathematically grounded solution. These spaces enable low-distortion embeddings of hierarchical structures (e.g., trees, taxonomies) and power-law distributions with substantially fewer dimensions compared to Euclidean counterparts. Recent advances have leveraged these properties to enhance foundation models, including improving LLMs' complex reasoning ability, VLMs' zero-shot generalization, and cross-modal semantic alignment, while maintaining parameter efficiency. This paper provides a comprehensive review of hyperbolic neural networks and their recent development for foundation models. We further outline key challenges and research directions to advance the field.

Citations

• HyperCore: The Core Framework for Building Hyperbolic Foundation Models with Comprehensive Modules
• Position: Beyond Euclidean -- Foundation Models Should Embrace Non-Euclidean Geometries
• GraphMoRE: Mitigating Topological Heterogeneity via Mixture of Riemannian Experts
• Round and Round We Go! What makes Rotary Positional Encodings useful?
• HVT: A Comprehensive Vision Framework for Learning in Non-Euclidean Space
• Hypformer: Exploring Efficient Transformer Fully in Hyperbolic Space
• Hardness of Learning Neural Networks under the Manifold Hypothesis
• Hyperbolic Geometric Latent Diffusion Model for Graph Generation
• From Local to Global: A Graph RAG Approach to Query-Focused Summarization
• The Revolution of Multimodal Large Language Models: A Survey
• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
• Hyperbolic Code Retrieval: A Novel Approach for Efficient Code Search Using Hyperbolic Space Embeddings
• Hyperbolic Image-Text Representations
• Vision-Language Models for Vision Tasks: A Survey
• Vision-Language Models for Vision Tasks: A Survey
• Hyperbolic Contrastive Learning for Visual Representations beyond Objects
• The Euclidean Space is Evil: Hyperbolic Attribute Editing for Few-shot Image Generation
• Diffusion Models in Vision: A Survey
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
• Interpretable RNA Foundation Model from Unannotated Data for Highly Accurate RNA Structure and Function Predictions
• Training Compute-Optimal Large Language Models
• Hyperbolic Vision Transformers: Combining Improvements in Metric Learning
• Modeling Heterogeneous Hierarchies with Relation-specific Hyperbolic Cones
• On the Opportunities and Risks of Foundation Models
• Modeling Scale-free Graphs with Hyperbolic Geometry for Knowledge-aware Recommendation
• Unsupervised Object-Level Representation Learning from Scene Images
• Fully Hyperbolic Neural Networks
• RoFormer: Enhanced Transformer with Rotary Position Embedding
• Lorentzian Graph Convolutional Networks
• A Hyperbolic-to-Hyperbolic Graph Convolutional Network
• Learning Transferable Visual Models From Natural Language Supervision
• Hyperbolic Deep Neural Networks: A Survey
• Score-Based Generative Modeling through Stochastic Differential Equations
• Dense Contrastive Learning for Self-Supervised Visual Pre-Training
• Graph Geometry Interaction Learning
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Simple and Deep Graph Convolutional Networks
• Adversarial Self-Supervised Contrastive Learning
• Language Models are Few-Shot Learners
• Geoopt: Riemannian Optimization in PyTorch
• Improved Baselines with Momentum Contrastive Learning
• Differentiating through the Fréchet Mean
• Latent Variable Modelling with Hyperbolic Normalizing Flows
• Hyperbolic Graph Attention Network
• Hyperbolic Graph Neural Networks
• Hyperbolic Graph Convolutional Neural Networks
• Visualizing and Measuring the Geometry of BERT
• Inferring Concept Hierarchies from Text Corpora via Hyperbolic Embeddings
• Continuous Hierarchical Representations with Poincar 'e Variational\n Auto-Encoders
• Poincaré GloVe: Hyperbolic Word Embeddings
• Learning Continuous Hierarchies in the Lorentz Model of Hyperbolic Geometry
• Hyperbolic Attention Networks
• Hyperbolic Neural Networks
• geomstats: a Python Package for Riemannian Geometry in Machine Learning
• Representation Tradeoffs for Hyperbolic Embeddings
• Hyperbolic Entailment Cones for Learning Hierarchical Embeddings
• Attention Is All You Need
• Poincaré Embeddings for Learning Hierarchical Representations
• Layer Normalization
• An Introduction to Convolutional Neural Networks
• Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
• Intrinsic Statistics on Riemannian Manifolds: Basic Tools for Geometric Measurements
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• The Gate Always Closes: On Injecting Auxiliary Signals into Frozen Vision-Language Models
• HyperbolicRAG: Enhancing Retrieval-Augmented Generation with Hyperbolic Representations
• Angular Gradient Sign Method: Uncovering Vulnerabilities in Hyperbolic Networks
• More Than A Shortcut: A Hyperbolic Approach To Early-Exit Networks
• Randomized HyperSteiner: A Stochastic Delaunay Triangulation Heuristic for the Hyperbolic Steiner Minimal Tree

Related

• Hyperbolic Deep Neural Networks: A Survey
• Hyperbolic Large Language Models
• HyperCore: The Core Framework for Building Hyperbolic Foundation Models with Comprehensive Modules
• Hyperbolic Convolutional Neural Networks
• HELM: Hyperbolic Large Language Models via Mixture-of-Curvature Experts