Unsupervised Document and Template Clustering using Multimodal Embeddings

2025/06/13 by Sampaio, Phillipe R., Maxcici, Helene · 1 citation
#Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Computer Vision and Pattern Recognition (cs.CV) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2506.12116

Abstract

We study unsupervised clustering of documents at both the category and template levels using frozen multimodal encoders and classical clustering algorithms. We systematize a model-agnostic pipeline that (i) projects heterogeneous last-layer states from text-layout-vision encoders into token-type-aware document vectors and (ii) performs clustering with centroid- or density-based methods, including an HDBSCAN + k-NN assignment to eliminate unlabeled points. We evaluate eight encoders (text-only, layout-aware, vision-only, and vision-language) with k-Means, DBSCAN, HDBSCAN + k-NN, and BIRCH on five corpora spanning clean synthetic invoices, their heavily degraded print-and-scan counterparts, scanned receipts, and real identity and certificate documents. The study reveals modality-specific failure modes and a robustness-accuracy trade-off, with vision features nearly solving template discovery on clean pages while text dominates under covariate shift, and fused encoders offering the best balance. We detail a reproducible, oracle-free tuning protocol and the curated evaluation settings to guide future work on unsupervised document organization.

Citations

• InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models
• Gemma 3 Technical Report
• LiGT: Layout-infused Generative Transformer for Visual Question Answering on Vietnamese Receipts
• Summaries as Centroids for Interpretable and Scalable Text Clustering
• Qwen2.5 Technical Report
• ColPali: Efficient Document Retrieval with Vision Language Models
• HRVDA: High-Resolution Visual Document Assistant
• KOSMOS-2.5: A Multimodal Literate Model
• Sigmoid Loss for Language Image Pre-Training
• Unifying Vision, Text, and Layout for Universal Document Processing
• Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks
• LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking
• DiT: Self-supervised Pre-training for Document Image Transformer
• OCR-free Document Understanding Transformer
• TrOCR: Transformer-based Optical Character Recognition with Pre-trained Models
• Representation Learning for Short Text Clustering
• DocFormer: End-to-End Transformer for Document Understanding
• Going Full-TILT Boogie on Document Understanding with Text-Image-Layout\n Transformer
• LayoutLMv2: Multi-modal Pre-training for Visually-Rich Document Understanding
• Towards a Multi-modal, Multi-task Learning based Pre-training Framework for Document Representation Learning
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• Shared Generative Latent Representation Learning for Multi-view Clustering
• DocBERT: BERT for Document Classification
• Multi-view low-rank sparse subspace clustering
• Unsupervised Deep Embedding for Clustering Analysis
• Return of Frustratingly Easy Domain Adaptation
• An End-to-End Trainable Neural Network for Image-based Sequence Recognition and Its Application to Scene Text Recognition
• Discriminatory Analysis. Nonparametric Discrimination: Consistency Properties
• Silhouettes: A graphical aid to the interpretation and validation of cluster analysis
• Least squares quantization in PCM
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Cited by

• IDP AutoOpt: Agent-Driven Optimization of Document Processing Pipeline Configurations

Related

• Semantic Term "Blurring" and Stochastic "Barcoding" for Improved Unsupervised Text Classification
• Resampling methods for document clustering
• UnSupDLA: Towards Unsupervised Document Layout Analysis
• Unsupervised Document Embedding With CNNs
• ClusterTabNet: Supervised clustering method for table detection and table structure recognition