Hierarchy-Aware Fine-Tuning of Vision-Language Models

2025/12/25 by Jiayu Li, Rajesh Gangireddy, Li, Jiayu +7
Computer Science · #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Topic Modeling

paper · doi:10.48550/arxiv.2512.21529

openalex publication_date 2025/12/25 · openalex created_date 2025/12/30 · openalex updated_date 2026/07/28

Abstract

Vision-Language Models (VLMs) learn powerful multimodal representations through large-scale image-text pretraining, but adapting them to hierarchical classification is underexplored. Standard approaches treat labels as flat categories and require full fine-tuning, which is expensive and produces inconsistent predictions across taxonomy levels. We propose an efficient hierarchy-aware fine-tuning framework that updates a few parameters while enforcing structural consistency. We combine two objectives: Tree-Path KL Divergence (TP-KL) aligns predictions along the ground-truth label path for vertical coherence, while Hierarchy-Sibling Smoothed Cross-Entropy (HiSCE) encourages consistent predictions among sibling classes. Both losses work in the VLM's shared embedding space and integrate with lightweight LoRA adaptation. Experiments across multiple benchmarks show consistent improvements in Full-Path Accuracy and Tree-based Inconsistency Error with minimal parameter overhead. Our approach provides an efficient strategy for adapting VLMs to structured taxonomies.

Citations

• Vision-Language Models for Edge Networks: A Comprehensive Survey
• CityLLaVA: Efficient Fine-Tuning for VLMs in City Scenario
• Cross-Modal Adapter: Parameter-Efficient Transfer Learning Approach for Vision-Language Models
• Revisiting Few-Shot Object Detection with Vision-Language Models
• VL-PET: Vision-and-Language Parameter-Efficient Tuning via Granularity Control
• Open-Ended Medical Visual Question Answering Through Prefix Tuning of Language Models
• UniAdapter: Unified Parameter-Efficient Transfer Learning for Cross-modal Modeling
• Hierarchical classification at multiple operating points
• EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning
• CoCa: Contrastive Captioners are Image-Text Foundation Models
• Vision-Language Pre-Training with Triple Contrastive Learning
• BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation
• A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models
• Align before Fuse: Vision and Language Representation Learning with Momentum Distillation
• MMF: Multi-Task Multi-Structure Fusion for Hierarchical Image Classification
• LoRA: Low-Rank Adaptation of Large Language Models
• The Power of Scale for Parameter-Efficient Prompt Tuning
• Learning Transferable Visual Models From Natural Language Supervision
• Training data-efficient image transformers & distillation through attention
• Language Models are Few-Shot Learners
• Hierarchical Deep Convolutional Neural Networks for Multi-category\n Diagnosis of Gastrointestinal Disorders on Histopathological Images
• Optuna: A Next-generation Hyperparameter Optimization Framework
• Parameter-Efficient Transfer Learning for NLP
• Fine-Grained Representation Learning and Recognition by Exploiting Hierarchical Semantic Embedding
• B-CNN: Branch Convolutional Neural Network for Hierarchical Classification
• Distilling the Knowledge in a Neural Network
• Fine-Grained Visual Classification of Aircraft

Related

• Fine-Grained VLM Fine-tuning via Latent Hierarchical Adapter Learning
• Hier-COS: Making Deep Features Hierarchy-aware via Composition of Orthogonal Subspaces
• Emergent Visual-Semantic Hierarchies in Image-Text Representations
• Rethinking Fine-Tuning: Unlocking Hidden Capabilities in Vision-Language Models
• Dynamic Embedding of Hierarchical Visual Features for Efficient Vision-Language Fine-Tuning