2024/11/08 by Laure Ciernik, Ciernik, Laure, Lorenz Linhardt +9 · 2 voices · 6 citations
Biochemistry, Genetics and Molecular Biology · Computer Science · Psychology · #Artificial intelligence #Cell Image Analysis Techniques #Computer science #Consistency (knowledge bases) #Data Visualization and Analytics #Machine learning #Neural Networks and Applications #Psychology #Similarity (geometry) #cs.AI #cs.CV #cs.LG
paper · pdf · doi:10.48550/arxiv.2411.05561
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2024/11/08 · openalex created_date 2024/11/15 · openalex updated_date 2026/08/05
The Platonic Representation Hypothesis claims that recent foundation models are converging to a shared representation space as a function of their downstream task performance, irrespective of the objectives and data modalities used to train these models (Huh et al., 2024). Representational similarity is generally measured for individual datasets and is not necessarily consistent across datasets. Thus, one may wonder whether this convergence of model representations is confounded by the datasets commonly used in machine learning. Here, we propose a systematic way to measure how representational similarity between models varies with the set of stimuli used to construct the representations. We find that the objective function is a crucial factor in determining the consistency of representational similarities across datasets. Specifically, self-supervised vision models learn representations whose relative pairwise similarities generalize better from one dataset to another compared to those of image classification or image-text models. Moreover, the correspondence between representational similarities and the models' task behavior is dataset-dependent, being most strongly pronounced for single-domain datasets. Our work provides a framework for analyzing similarities of model representations across datasets and linking those similarities to differences in task behavior.