Hypergraph Vision Transformers: Images are More than Nodes, More than Edges

2025/04/11 by Joshua Fixelle, Fixelle, Joshua · 8 citations
Computer Science · Engineering · #Advanced Neural Network Applications #CCD and CMOS Imaging Sensors #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Visual Attention and Saliency Detection

paper · pdf · doi:10.48550/arxiv.2504.08710

openalex publication_date 2025/04/11 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent advancements in computer vision have highlighted the scalability of Vision Transformers (ViTs) across various tasks, yet challenges remain in balancing adaptability, computational efficiency, and the ability to model higher-order relationships. Vision Graph Neural Networks (ViGs) offer an alternative by leveraging graph-based methodologies but are hindered by the computational bottlenecks of clustering algorithms used for edge generation. To address these issues, we propose the Hypergraph Vision Transformer (HgVT), which incorporates a hierarchical bipartite hypergraph structure into the vision transformer framework to capture higher-order semantic relationships while maintaining computational efficiency. HgVT leverages population and diversity regularization for dynamic hypergraph construction without clustering, and expert edge pooling to enhance semantic extraction and facilitate graph-based image retrieval. Empirical results demonstrate that HgVT achieves strong performance on image classification and retrieval, positioning it as an efficient framework for semantic-based vision tasks.

Cited by

• Commonality in Few: Few-Shot Multimodal Anomaly Detection via Hypergraph-Enhanced Memory
• Hyper-FEOD: Sparse Hypergraph-Enhanced Frame-Event Object Detection with Fine-Grained MoE
• AttentionViG: Cross-Attention-Based Dynamic Neighbor Aggregation in Vision GNNs
• DHG-Bench: A Comprehensive Benchmark for Deep Hypergraph Learning
• HGTS-Former: Hierarchical HyperGraph Transformer for Multivariate Time Series Analysis
• Hybrid Hypergraph Networks for Multimodal Sequence Data Classification
• YOLOv13: Real-Time Object Detection with Hypergraph-Enhanced Adaptive Visual Perception
• SoftHGNN: Soft Hypergraph Neural Networks for General Visual Recognition

Related

• HGFormer: Topology-Aware Vision Transformer with HyperGraph Learning
• GvT: A Graph-based Vision Transformer with Talking-Heads Utilizing Sparsity, Trained from Scratch on Small Datasets
• A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspective
• DVHGNN: Multi-Scale Dilated Vision HGNN for Efficient Vision Recognition
• SAG-ViT: A Scale-Aware, High-Fidelity Patching Approach with Graph Attention for Vision Transformers