Understanding BLOOM: An empirical study on diverse NLP tasks

2022/11/27 by Dakle, Parag Pravin, Rallabandi, SaiKrishna, Raghavan, Preethi
#Computation and Language (cs.CL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2211.14865

Abstract

We view the landscape of large language models (LLMs) through the lens of the recently released BLOOM model to understand the performance of BLOOM and other decoder-only LLMs compared to BERT-style encoder-only models. We achieve this by evaluating the smaller BLOOM model variants (350m/560m and 1b3/1b7) on several NLP benchmark datasets and popular leaderboards. We make the following observations: (1) BLOOM performance does not scale with parameter size, unlike other LLMs like GPT and BERT. Experiments fine-tuning BLOOM models show that the 560m variant performs similarly to or better than the 1b7 variant, (2) Zero-shot cross-lingual and multi-lingual fine-tuning experiments show that BLOOM is at par or worse than monolingual GPT-2 models, and (3) Toxicity analysis of prompt-based text generation using the RealToxicityPrompts dataset shows that the text generated by BLOOM is at least 17% less toxic than GPT-2 and GPT-3 models.

Related

• Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM
• BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
• LACoS-BLOOM: Low-rank Adaptation with Contrastive objective on 8 bits Siamese-BLOOM
• Comparing Specialised Small and General Large Language Models on Text Classification: 100 Labelled Samples to Achieve Break-Even Performance
• Large Language Models: A Survey