Towards an Explainable Comparison and Alignment of Feature Embeddings

2025/06/06 by Mohammad S. Jalali, Jalali, Mohammad, Bahar Dibaei Nia +3 · 1 citation
Computer Science · #Advanced Graph Neural Networks #Artificial Intelligence (cs.AI) #Computer Vision and Pattern Recognition (cs.CV) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #FOS: Mathematics #Face and Expression Recognition #Machine Learning (cs.LG) #Spectral Theory (math.SP)

paper · pdf · doi:10.48550/arxiv.2506.06231

openalex publication_date 2025/06/06 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

While several feature embedding models have been developed in the literature, comparisons of these embeddings have largely focused on their numerical performance in classification-related downstream applications. However, an interpretable comparison of different embeddings requires identifying and analyzing mismatches between sample groups clustered within the embedding spaces. In this work, we propose the Spectral Pairwise Embedding Comparison (SPEC) framework to compare embeddings and identify their differences in clustering a reference dataset. Our approach examines the kernel matrices derived from two embeddings and leverages the eigendecomposition of the difference kernel matrix to detect sample clusters that are captured differently by the two embeddings. We present a scalable implementation of this kernel-based approach, with computational complexity that grows linearly with the sample size. Furthermore, we introduce an optimization problem using this framework to align two embeddings, ensuring that clusters identified in one embedding are also captured in the other model. We provide numerical results demonstrating the SPEC's application to compare and align embeddings on large-scale datasets such as ImageNet and MS-COCO. The project page is available at https://mjalali.github.io/SPEC/.

Citations

• Kernel-based Unsupervised Embedding Alignment for Enhanced Visual Representation in Vision-language Models
• PromptWise: Online Learning for Cost-Aware Prompt Assignment in Generative Models
• Scendi Score: Prompt-Aware Diversity Evaluation via Schur Complement of CLIP Embeddings
• Do Vendi Scores Converge with Finite Samples? Truncated Vendi Score for Finite-Sample Convergence Guarantees
• PAK-UCB Contextual Bandit: An Online Learning Approach to Prompt-Aware Selection of Generative Models and LLMs
• SEA: Supervised Embedding Alignment for Token-Level Visual-Textual Integration in MLLMs
• Understanding Generative AI Content with Embedding Models
• ModalChorus: Visual Probing and Alignment of Multi-modal Embeddings via Modal Fusion Map
• Towards a Scalable Reference-Free Evaluation of Generative Models
• Unaligning Everything: Or Aligning Any Text to Any Image in Multimodal Models
• A Multi-Armed Bandit Approach to Online Selection and Evaluation of Generative Models
• When is an Embedding Model More Promising than Another?
• Unveiling Differences in Generative Models: A Scalable Differential Clustering Approach
• An Interpretable Evaluation of Entropy-based Novelty of Generative Models
• Improving Text Embeddings with Large Language Models
• OneLLM: One Framework to Align All Modalities with Language
• Cousins Of The Vendi Score: A Family Of Similarity-Based Diversity Metrics For Science And Machine Learning
• On the Distributed Evaluation of Generative Models
• MultiFusion: Fusing Pre-Trained Models for Multi-Lingual, Multi-Modal Image Generation
• Connecting Multi-modal Contrastive Representations
• ImageBind: One Embedding Space To Bind Them All
• Visual Instruction Tuning
• Analyzing Transformers in Embedding Space
• Interpreting Embedding Spaces by Conceptualization
• Disentangling visual and written concepts in CLIP
• Flamingo: a Visual Language Model for Few-Shot Learning
• The Role of ImageNet Classes in Fréchet Inception Distance
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• SimCSE: Simple Contrastive Learning of Sentence Embeddings
• Learning Transferable Visual Models From Natural Language Supervision
• Probing Classifiers: Promises, Shortcomings, and Advances
• Evaluation of BERT and ALBERT Sentence Embedding Performance on Downstream NLP Tasks
• Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
• Information-Theoretic Probing for Linguistic Structure
• A Primer in BERTology: What we know about how BERT works
• Embedding Comparator: Visualizing Differences in Global Structure and Local Neighborhoods via Small Multiples
• StarGAN v2: Diverse Image Synthesis for Multiple Domains
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• A Style-Based Generator Architecture for Generative Adversarial Networks
• Streaming Kernel PCA with O(√(n)) Random Features
• Evaluation of sentence embeddings in downstream and linguistic probing\n tasks
• Unsupervised Alignment of Embeddings with Wasserstein Procrustes
• UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction
• Demystifying MMD GANs
• Sub-sampled Cubic Regularization for Non-convex Optimization
• Pointer Sentinel Mixture Models
• Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks
• Streaming Kernel Principal Component Analysis
• Rethinking the Inception Architecture for Computer Vision
• On the Error of Random Fourier Features
• The Expressive Power of Word Embeddings
• Conditional Vendi Score: An Information-Theoretic Approach to Diversity Evaluation of Prompt-based Generative Models
• Feature Likelihood Divergence: Evaluating the Generalization of Generative Models Using Samples

Cited by

• SPARKE: Scalable Prompt-Aware Diversity and Novelty Guidance in Diffusion Models via RKE Score

Related

• Spectral Embedding Norm: Looking Deep into the Spectrum of the Graph Laplacian
• Emblaze: Illuminating Machine Learning Representations through Interactive Comparison of Embedding Spaces
• Embedding Comparator: Visualizing Differences in Global Structure and Local Neighborhoods via Small Multiples
• When Kernels Multiply, Clusters Unify: Fusing Embeddings with the Kronecker Product
• Co-Embedding: Discovering Communities on Bipartite Graphs through Projection