JECL: Joint Embedding and Cluster Learning for Image-Text Pairs

2019/01/04 by Yang, Sean T., Huang, Kuan-Hao, Howe, Bill · 1 citation
#FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML)

paper · doi:10.48550/arxiv.1901.01860

Abstract

We propose JECL, a method for clustering image-caption pairs by training parallel encoders with regularized clustering and alignment objectives, simultaneously learning both representations and cluster assignments. These image-caption pairs arise frequently in high-value applications where structured training data is expensive to produce, but free-text descriptions are common. JECL trains by minimizing the Kullback-Leibler divergence between the distribution of the images and text to that of a combined joint target distribution and optimizing the Jensen-Shannon divergence between the soft cluster assignments of the images and text. Regularizers are also applied to JECL to prevent trivial solutions. Experiments show that JECL outperforms both single-view and multi-view methods on large benchmark image-caption datasets, and is remarkably robust to missing captions and varying data sizes.

Cited by

• Fragments to Facts: Partial-Information Fragment Inference from LLMs

Related

• Image Clustering via the Principle of Rate Reduction in the Age of Pretrained Models
• Keep It Light! Simplifying Image Clustering Via Text-Free Adapters
• Clustering by Maximizing Mutual Information Across Views
• Self-Enhanced Image Clustering with Cross-Modal Semantic Consistency
• Deep Clustering via Joint Convolutional Autoencoder Embedding and Relative Entropy Minimization