Language-driven Fine-grained Retrieval

2025/12/06 by Wang, Shijie, Yu, Xin, Luo, Yadan +3
#Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2512.06255

Abstract

Existing fine-grained image retrieval (FGIR) methods learn discriminative embeddings by adopting semantically sparse one-hot labels derived from category names as supervision. While effective on seen classes, such supervision overlooks the rich semantics encoded in category names, hindering the modeling of comparability among cross-category details and, in turn, limiting generalization to unseen categories. To tackle this, we introduce LaFG, a Language-driven framework for Fine-Grained Retrieval that converts class names into attribute-level supervision using large language models (LLMs) and vision-language models (VLMs). Treating each name as a semantic anchor, LaFG prompts an LLM to generate detailed, attribute-oriented descriptions. To mitigate attribute omission in these descriptions, it leverages a frozen VLM to project them into a vision-aligned space, clustering them into a dataset-wide attribute vocabulary while harvesting complementary attributes from related categories. Leveraging this vocabulary, a global prompt template selects category-relevant attributes, which are aggregated into category-specific linguistic prototypes. These prototypes supervise the retrieval model to steer

Citations

• ProKeR: A Kernel Perspective on Few-Shot Adaptation of Large Vision-Language Models
• Enhancing Visual Continual Learning with Language-Guided Supervision
• Language-guided Image Reflection Separation
• Learning Semantic Proxies from Visual Prompts for Parameter-Efficient Fine-Tuning in Deep Metric Learning
• ChatQA: Surpassing GPT-4 on Conversational QA and RAG
• The Dawn After the Dark: An Empirical Study on Factuality Hallucination in Large Language Models
• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
• Tuning Multi-mode Token-level Prompt Alignment across Modalities
• Introspective Deep Metric Learning
• CLIP-S⁴: Language-Guided Self-Supervised Semantic Segmentation
• GPT-4 Technical Report
• HIER: Metric Learning Beyond Class Labels via Hierarchical Regularization
• Towards Reasoning in Large Language Models: A Survey
• Contrastive Bayesian Analysis for Deep Metric Learning
• Hyperbolic Vision Transformers: Combining Improvements in Metric Learning
• Non-isotropy Regularization for Proxy-based Deep Metric Learning
• MVPTR: Multi-Level Semantic Alignment for Vision-Language Pre-Training via Multi-Stage Learning
• FILIP: Fine-grained Interactive Language-Image Pre-Training
• Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm
• Deep Relational Metric Learning
• Learning by Planning: Language-Guided Global Image Editing
• Learning with Memory-based Virtual Classes for Deep Metric Learning
• Embedding Transfer with Label Relaxation for Improved Metric Learning
• Learning Transferable Visual Models From Natural Language Supervision
• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale
• Keypoint-Aligned Embeddings for Image Retrieval and Re-identification
• Learning Visual Representations with Caption Annotations
• VirTex: Learning Visual Representations from Textual Annotations
• Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks
• ProxyNCA++: Revisiting and Revitalizing Proxy Neighborhood Component\n Analysis
• LXMERT: Learning Cross-Modality Encoder Representations from Transformers
• Link the head to the "beak": Zero Shot Learning from Noisy Text Description at Part Precision
• Selective Convolutional Descriptor Aggregation for Fine-Grained Image Retrieval
• Deep Residual Learning for Image Recognition
• Deep Metric Learning via Lifted Structured Feature Embedding
• Bird Species Categorization Using Pose Normalized Deep Convolutional Nets

Related

• Democratizing Fine-grained Visual Recognition with Large Language Models
• Thinking Beyond Labels: Vocabulary-Free Fine-Grained Recognition using Reasoning-Augmented LMMs
• Rethinking Sparse Lexical Representations for Image Retrieval in the Age of Rising Multi-Modal Large Language Models
• Visual Classification via Description from Large Language Models
• Learning to Name Classes for Vision and Language Models