OTTER: Open-Tagging via Text-Image Representation for Multi-modal Understanding

2025/10/01 by Jieer Ouyang, Xiaoneng Xiang, Ouyang, Jieer +5
Computer Science · #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2510.00652

openalex publication_date 2025/10/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We introduce OTTER, a unified open-set multi-label tagging framework that harmonizes the stability of a curated, predefined category set with the adaptability of user-driven open tags. OTTER is built upon a large-scale, hierarchically organized multi-modal dataset, collected from diverse online repositories and annotated through a hybrid pipeline combining automated vision-language labeling with human refinement. By leveraging a multi-head attention architecture, OTTER jointly aligns visual and textual representations with both fixed and open-set label embeddings, enabling dynamic and semantically consistent tagging. OTTER consistently outperforms competitive baselines on two benchmark datasets: it achieves an overall F1 score of 0.81 on Otter and 0.75 on Favorite, surpassing the next-best results by margins of 0.10 and 0.02, respectively. OTTER attains near-perfect performance on open-set labels, with F1 of 0.99 on Otter and 0.97 on Favorite, while maintaining competitive accuracy on predefined labels. These results demonstrate OTTER's effectiveness in bridging closed-set consistency with open-vocabulary flexibility for multi-modal tagging applications.

Citations

• SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features
• Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution
• MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicine
• Data Processing Techniques for Modern Multimodal Models
• Multimodal Multilabel Classification by CLIP
• What Do You See? Enhancing Zero-Shot Image Classification with Multimodal Large Language Models
• OneLLM: One Framework to Align All Modalities with Language
• CLIP-Decoder : ZeroShot Multilabel Classification using Multimodal CLIP Aligned Representation
• Open Vocabulary Multi-Label Classification with Dual-Modal Decoder on Aligned Visual-Textual Features
• Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification
• ML-Decoder: Scalable and Versatile Classification Head
• Attention-Driven Dynamic Graph Convolutional Network for Multi-Label Image Recognition
• Cross-Modality Attention with Semantic Graph Embedding for Multi-Label Classification
• Learning Semantic-Specific Graph Representation for Multi-Label Image Recognition
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• What Makes Training Multi-Modal Classification Networks Hard?
• Bilinear Attention Networks
• Multi-label Image Recognition by Recurrently Discovering Attentional Regions
• SGDR: Stochastic Gradient Descent with Warm Restarts
• Exploit Bounding Box Annotations for Multi-label Object Recognition
• Zero-Shot Learning by Convex Combination of Semantic Embeddings
• Image Labeling on a Network: Using Social-Network Metadata for Image Classification

Related

• Open-Set Image Tagging with Multi-Grained Text Supervision
• Recognize Anything: A Strong Image Tagging Model
• Text as Image: Learning Transferable Adapter for Multi-Label Classification
• Data-free Multi-label Image Recognition via LLM-powered Prompt Tuning
• Category-Adaptive Cross-Modal Semantic Refinement and Transfer for Open-Vocabulary Multi-Label Recognition