Learning Concept Taxonomies from Multi-modal Data

2016/06/29 by Hao Zhang, Zhang, Hao, Zhiting Hu +9 · 1 citation
Computer Science · #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences #Image Retrieval and Classification Techniques #Machine Learning (cs.LG) #Multimodal Machine Learning Applications #Video Analysis and Summarization

paper · pdf · doi:10.48550/arxiv.1606.09239

openalex publication_date 2016/06/29 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We study the problem of automatically building hypernym taxonomies from textual and visual data. Previous works in taxonomy induction generally ignore the increasingly prominent visual data, which encode important perceptual semantics. Instead, we propose a probabilistic model for taxonomy induction by jointly leveraging text and images. To avoid hand-crafted feature engineering, we design end-to-end features based on distributed representations of images and words. The model is discriminatively trained given a small set of existing ontologies and is capable of building full taxonomies from scratch for a collection of unseen conceptual label items with associated images. We evaluate our model and features on the WordNet hierarchies, where our system outperforms previous approaches by a large gap.

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition

Cited by

• Contrastive Learning with Enhanced Abstract Representations using Grouped Loss of Abstract Semantic Supervision

Related

• Taxonomy Induction using Hypernym Subsequences
• Learning Taxonomies of Concepts and not Words using Contextualized Word Representations: A Position Paper
• TaBIIC: Taxonomy Building through Iterative and Interactive Clustering
• End-to-End Reinforcement Learning for Automatic Taxonomy Induction
• Every child should have parents: a taxonomy refinement algorithm based on hyperbolic term embeddings