OS-HGAdapter: Open Semantic Hypergraph Adapter for Large Language Models Assisted Entropy-Enhanced Image-Text Alignment

2025/10/15 by Rongjun Chen, Chengsi Yao, Chen, Rongjun +15
Computer Science · Social Sciences · #Computational and Text Analysis Methods #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimedia (cs.MM) #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2510.13131

openalex publication_date 2025/10/15 · openalex created_date 2025/10/17 · openalex updated_date 2026/07/28

Abstract

Text-image alignment constitutes a foundational challenge in multimedia content understanding, where effective modeling of cross-modal semantic correspondences critically enhances retrieval system performance through joint embedding space optimization. Given the inherent difference in information entropy between texts and images, conventional approaches often show an imbalance in the mutual retrieval of these two modalities. To address this particular challenge, we propose to use the open semantic knowledge of Large Language Model (LLM) to fill for the entropy gap and reproduce the alignment ability of humans in these tasks. Our entropy-enhancing alignment is achieved through a two-step process: 1) a new prompt template that does not rely on explicit knowledge in the task domain is designed to use LLM to enhance the polysemy description of the text modality. By analogy, the information entropy of the text modality relative to the visual modality is increased; 2) A hypergraph adapter is used to construct multilateral connections between the text and image modalities, which can correct the positive and negative matching errors for synonymous semantics in the same fixed embedding space, whilst reducing the noise caused by open semantic entropy by mapping the reduced dimensions back to the original dimensions. Comprehensive evaluations on the Flickr30K and MS-COCO benchmarks validate the superiority of our Open Semantic Hypergraph Adapter (OS-HGAdapter), showcasing 16.8% (text-to-image) and 40.1% (image-to-text) cross-modal retrieval gains over existing methods while establishing new state-of-the-art performance in semantic alignment tasks.

Citations

• Composing Object Relations and Attributes for Image-Text Matching
• Improved Probabilistic Image-Text Representations
• PMR: Prototypical Modal Rebalance for Multimodal Learning
• What does a platypus look like? Generating customized prompts for zero-shot image classification
• Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks
• Learning in Audio-visual Context: A Review, Analysis, and New Perspective
• HGNN + : General Hypergraph Neural Networks
• ECCV Caption: Correcting False Negatives by Collecting Machine-and-Human-verified Image-Caption Associations for MS-COCO
• Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)
• Vision-Language Pre-Training with Triple Contrastive Learning
• Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks
• CLIP-Adapter: Better Vision-Language Models with Feature Adapters
• CLIP-Adapter: Better Vision-Language Models with Feature Adapters
• M5Product: Self-harmonized Contrastive Learning for E-commercial Multi-modal Pretraining
• Learning to Prompt for Vision-Language Models
• LoRA: Low-Rank Adaptation of Large Language Models
• BEiT: BERT Pre-Training of Image Transformers
• Cross-Modal Discrete Representation Learning
• Learning Transferable Visual Models From Natural Language Supervision
• Probabilistic Embeddings for Cross-Modal Retrieval
• Similarity Reasoning and Filtration for Image-Text Matching
• Learning the Best Pooling Strategy for Visual Semantic Embedding
• Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO
• How Can We Know What Language Models Know?
• Cross-modal Scene Graph Matching for Relationship-aware Image-Text Retrieval
• Focus Your Attention: A Bidirectional Focal Attention Network for\n Image-Text Matching
• Visual Semantic Reasoning for Image-Text Matching
• Language Models as Knowledge Bases?
• Hypergraph Neural Networks
• Stacked Cross Attention for Image-Text Matching
• Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering
• Attention Is All You Need
• Instance-aware Image and Sentence Matching with Selective Multimodal LSTM
• Semi-Supervised Classification with Graph Convolutional Networks
• Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations
• Order-Embeddings of Images and Language
• Particular object retrieval with integral max-pooling of CNN activations
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal\n Networks
• Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks
• Microsoft COCO Captions: Data Collection and Evaluation Server
• Stochastic Pooling for Regularization of Deep Convolutional Neural Networks
• Normalized Mutual Information to evaluate overlapping community finding algorithms
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• MultiWay-Adapater: Adapting large-scale multi-modal models for scalable image-text retrieval
• Fine-grained Visual Textual Alignment for Cross-Modal Retrieval using\n Transformer Encoders
• VISTA: Enhancing Vision-Text Alignment in MLLMs via Cross-Modal Mutual Information Maximization
• CAMP: Cross-Modal Adaptive Message Passing for Text-Image Retrieval
• HiMo-CLIP: Modeling Semantic Hierarchy and Monotonicity in Vision-Language Alignment