Multimodal Search on Iconclass using Vision-Language Pre-Trained Models

2023/06/23 by Cristian Santini, Santini, Cristian, Etienne Posthumus +9 · 1 citation
Computer Science · Earth and Planetary Sciences · #Advanced Image and Video Retrieval Techniques #Archaeological Research and Protection #Artificial Intelligence (cs.AI) #Digital Libraries (cs.DL) #FOS: Computer and information sciences #Information Retrieval (cs.IR)

paper · pdf · doi:10.48550/arxiv.2306.16529

openalex publication_date 2023/06/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Terminology sources, such as controlled vocabularies, thesauri and classification systems, play a key role in digitizing cultural heritage. However, Information Retrieval (IR) systems that allow to query and explore these lexical resources often lack an adequate representation of the semantics behind the user's search, which can be conveyed through multiple expression modalities (e.g., images, keywords or textual descriptions). This paper presents the implementation of a new search engine for one of the most widely used iconography classification system, Iconclass. The novelty of this system is the use of a pre-trained vision-language model, namely CLIP, to retrieve and explore Iconclass concepts using visual or textual queries.

Cited by

• Automating Iconclass: LLMs and RAG for Large-Scale Classification of Religious Woodcuts

Related

• Enhancing Image Retrieval : A Comprehensive Study on Photo Search using the CLIP Mode
• Transferring AI-Based Iconclass Classification Across Image Traditions: A RAG Pipeline for the Wenzelsbibel
• Automating Iconclass: LLMs and RAG for Large-Scale Classification of Religious Woodcuts
• Benchmarking Vision-Language and Multimodal Large Language Models in Zero-shot and Few-shot Scenarios: A study on Christian Iconography
• RAVEN: Multitask Retrieval Augmented Vision-Language Learning