No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance

2024/04/04 by Vishaal Udandarao, Ameya Prabhu, Udandarao, Vishaal +13 · 15 voices · 10 citations
Computer Science · #Advanced Text Analysis Techniques

paper · pdf · doi:10.48550/arxiv.2404.04125

Abstract

Web-crawled pretraining datasets underlie the impressive "zero-shot" evaluation performance of multimodal models, such as CLIP for classification/retrieval and Stable-Diffusion for image generation. However, it is unclear how meaningful the notion of "zero-shot" generalization is for such multimodal models, as it is not known to what extent their pretraining datasets encompass the downstream concepts targeted for during "zero-shot" evaluation. In this work, we ask: How is the performance of multimodal models on downstream concepts influenced by the frequency of these concepts in their pretraining datasets? We comprehensively investigate this question across 34 models and five standard pretraining datasets (CC-3M, CC-12M, YFCC-15M, LAION-400M, LAION-Aesthetics), generating over 300GB of data artifacts. We consistently find that, far from exhibiting "zero-shot" generalization, multimodal models require exponentially more data to achieve linear improvements in downstream "zero-shot" performance, following a sample inefficient log-linear scaling trend. This trend persists even when controlling for sample-level similarity between pretraining and downstream datasets, and testing on purely synthetic data distributions. Furthermore, upon benchmarking models on long-tailed data sampled based on our analysis, we demonstrate that multimodal models across the board perform poorly. We contribute this long-tail test set as the "Let it Wag!" benchmark to further research in this direction. Taken together, our study reveals an exponential need for training data which implies that the key to "zero-shot" generalization capabilities under large-scale training paradigms remains to be found.

Cited by

• When Pretty Isn't Useful: Investigating Why Modern Text-to-Image Models Fail as Reliable Training Data Generators
• Sparse Autoencoders are Topic Models
• Not All Data Are Unlearned Equally
• Is Generation Required for Data-Efficient Perception?
• Selective Masking based Self-Supervised Learning for Image Semantic Segmentation
• How Bias Binds: Measuring Hidden Associations for Bias Control in Text-to-Image Compositions
• Data-Centric Lessons To Improve Speech-Language Pretraining
• Person-Centric Annotations of LAION-400M: Auditing Bias and Its Transfer to Models
• Review of Hallucination Understanding in Large Language and Vision Models
• Hallucinations in medical devices

Discussions

• No "Zero-Shot" Without Exponential Data [hn, 187 points, 118 comments]
• Has Generative AI Already Peaked? - Computerphile [lemmy, 85 points, 26 comments]
• Es ist echt schwer zu illustrieren, warum "Paper mit GPT zusammenfassen" im Allgemeinen keine gute Idee ist, ohne Fachwissen. Hab GPT 4o dieses Paper aus "meinem" Gebiet gefüttert, erschienen nach de [bsky, 30 points, 2 comments]
• 「我々の研究は、訓練データが指数関数的に必要であることを明らかにし、大規模な訓練パラダイムにおける『ゼロショット』汎化能力の鍵はまだ見つかっていないことを示唆する」金とGPUつぎ込んでも今のままではAGIには到達しないちゅうことか No "Zero-Shot" Without Exponential Data: Pretraining Concept Frequency Determines Mu [bsky, 3 points, 0 comments]
• arxiv.org/abs/2404.04125 currently, in order to make linear progress improving a model, the amount of training data must grow at an exponential rate, the industry is already consuming all data it can [bsky, 2 points, 2 comments]
• AI has an upper bound for how useful it can be with the amount of data is input (see attached study, released Apr this year). I think we're going to see AI plateau in effectiveness and users are going [bsky, 2 points, 0 comments]
• Very interesting, thanks. Here's another nail in the coffin (on fast diminishing returns as a function of training data size) arxiv.org/abs/2404.04125 [bsky, 2 points, 0 comments]
• Paper finds a log-linear relationship between concept frequency (nouns like man, hat, fish, etc.) and 0-shot performance: the model needs to see exponentially more examples of a concept in training to [bsky, 2 points, 2 comments]
• No "Zero-Shot" Without Exponential Data [hn, 2 points, 0 comments]
• No "Zero-Shot" Without Exponential Data [hn, 2 points, 0 comments]
• No "Zero-Shot" Without Exponential Data [hn, 1 points, 0 comments]
• 대용량 멀티모달 모델들의 "zero-shot" 성능이 실제로는 pretraining 데이터셋에서의 해당 개념의 빈도에 크게 영향을 받는다는 내용. 분류, 검색, 이미지 생성 등 다양한 downstream task에서 34개의 모델들을 테스트한 결과, pretraining 데이터에서 해당 개념의 로그 빈도와 zero-shot 성능 간에 선형 관계가 있음을 발 [bsky, 1 points, 1 comments]
• LLM AIs get better as the logarithm of the size of the training data (preprint): https://arxiv.org/abs/2404.04125 which means that getting to AGI is exponentially hard (as I said might be the case s [bsky, 0 points, 0 comments]
• No "Zero-Shot" Without Exponential Data (arxiv.org) Main Link | Discussion [bsky, 0 points, 0 comments]
• Attention is NOT everything. Linear performance gains require exponentially more data. Zero shot training is a mirage. Current AI architectures will never achieve the generalization required for AGI. [bsky, 0 points, 0 comments]

Related

• Towards Accelerated Model Training via Bayesian Data Selection
• Benchmarking Zero-Shot Robustness of Multimodal Foundation Models: A Pilot Study
• Zero-Shot Robustification of Zero-Shot Models
• VL-Taboo: An Analysis of Attribute-based Zero-shot Capabilities of Vision-Language Models
• The Neglected Tails in Vision-Language Models