BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
2022/11/09 by BigScience Workshop, Workshop, BigScience, Teven Le Scao +784 · 1 voice · 211 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #cs.CL
paper · pdf · doi:10.48550/arxiv.2211.05100
arxiv published 2022/11/09 · arxiv updated 2023/06/27
Abstract
Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. While these capabilities have led to widespread adoption, most LLMs are developed by resource-rich organizations and are frequently kept from the public. As a step towards democratizing this powerful technology, we present BLOOM, a 176B-parameter open-access language model designed and built thanks to a collaboration of hundreds of researchers. BLOOM is a decoder-only Transformer language model that was trained on the ROOTS corpus, a dataset comprising hundreds of sources in 46 natural and 13 programming languages (59 in total). We find that BLOOM achieves competitive performance on a wide variety of benchmarks, with stronger results after undergoing multitask prompted finetuning. To facilitate future research and applications using LLMs, we publicly release our models and code under the Responsible AI License.
Cited by
- Form and Meaning in Intrinsic Multilingual Evaluations
- Optimizing Resource Allocation for Geographically-Distributed Inference by Large Language Models
- TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior
- Designing Spatial Architectures for Sparse Attention: STAR Accelerator via Cross-Stage Tiling
- FAME: Fictional Actors for Multilingual Erasure
- Beyond Fast and Slow: Cognitive-Inspired Elastic Reasoning for Large Language Models
- PADE: A Predictor-Free Sparse Attention Accelerator via Unified Execution and Stage Fusion
- PrahokBART: A Pre-trained Sequence-to-Sequence Model for Khmer Natural Language Generation
- BOOST: BOttleneck-Optimized Scalable Training Framework for Low-Rank Large Language Models
- XDoGE: Multilingual Data Reweighting to Enhance Language Inclusivity in LLMs
- MASim: Multilingual Agent-Based Simulation for Social Science
- Are LLMs Truly Multilingual? Exploring Zero-Shot Multilingual Capability of LLMs for Information Retrieval: An Italian Healthcare Use Case
- AR-Med: Automated Relevance Enhancement in Medical Search via LLM-Driven Information Augmentation
- ESACT: An End-to-End Sparse Accelerator for Compute-Intensive Transformers via Local Similarity
- Evaluation of Large Language Models for Numeric Anomaly Detection in Power Systems
- TrackList: Tracing Back Query Linguistic Diversity for Head and Tail Knowledge in Open Large Language Models
- LAPA: Log-Domain Prediction-Driven Dynamic Sparsity Accelerator for Transformer Model
- Scaling Competence, Shrinking Reasoning: Cognitive Signatures in Language Model Learning
- Don't Think of the White Bear: Ironic Negation in Transformer Models Under Cognitive Load
- A Structure-Agnostic Co-Tuning Framework for LLMs and SLMs in Cloud-Edge Systems
- NOTAM-Evolve: A Knowledge-Guided Self-Evolving Optimization Framework with LLMs for NOTAM Interpretation
- Wasm: A Pipeline for Constructing Structured Arabic Interleaved Multimodal Corpora
- Ghost in the Transformer: Detecting Model Reuse with Invariant Spectral Signatures
- MIMIC-SR-ICD11: A Dataset for Narrative-Based Diagnosis
- From Model to Breach: Towards Actionable LLM-Generated Vulnerabilities Reporting
- Advancing Equitable AI: Evaluating Cultural Expressiveness in LLMs for Latin American Contexts
- Analyzing the Power of Chain of Thought through Memorization Capabilities
- How Different Tokenization Algorithms Impact LLMs and Transformer Models for Binary Code Analysis
- Dynamic Reflections: Probing Video Representations with Text Alignment
- LIR: The First Workshop on Late Interaction and Multi Vector Retrieval @ ECIR 2026
- Languages are Modalities: Cross-Lingual Alignment via Encoder Injection
- Encoder-Decoder or Decoder-Only? Revisiting Encoder-Decoder Large Language Model
- 1+1>2: A Synergistic Sparse and Low-Rank Compression Method for Large Language Models
- Evaluating LLMs on Generating Age-Appropriate Child-Like Conversations
- Fine-tuning of Large Language Models for Constituency Parsing Using a Sequence to Sequence Approach
- A Data-Centric Approach to Multilingual E-Commerce Product Search: Case Study on Query-Category and Query-Item Relevance
- Tibetan Language and AI: A Comprehensive Survey of Resources, Methods and Challenges
- From Local to Global: Revisiting Structured Pruning Paradigms for Large Language Models
- DETree: DEtecting Human-AI Collaborative Texts via Tree-Structured Hierarchical Representation Learning
- All You Need is One: Capsule Prompt Tuning with a Single Vector
- The German Commons - 154 Billion Tokens of Openly Licensed Text for German Language Models
- Document Intelligence in the Era of Large Language Models: A Survey
- Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanism
- Litespark Technical Report: High-Throughput, Energy-Efficient LLM Training Framework
- An Explorative Study on Distributed Computing Techniques in Training and Inference of Large Language Models
- Surgical Repair of Collapsed Attention Heads in ALiBi Transformers
- SynthID-Image: Image watermarking at internet scale
- Sunflower: A New Approach To Expanding Coverage of African Languages in Large Language Models
- Mid-Training of Large Language Models: A Survey
- Luth: Efficient French Specialization for Small Language Models and Cross-Lingual Transfer
- Mixture of Neuron Experts
- The African Languages Lab: A Collaborative Approach to Advancing Low-Resource African NLP
- The New Quant: A Survey of Large Language Models in Financial Prediction and Trading
- Parallax: Efficient LLM Inference Service over Decentralized Environment
- QFrBLiMP: a Quebec-French Benchmark of Linguistic Minimal Pairs
- Emergent evaluation hubs in a decentralizing large language model ecosystem
- Vision Function Layer in Multimodal LLMs
- PonderLM-2: Pretraining LLM with Latent Thoughts in Continuous Space
- Linear Causal Representation Learning by Topological Ordering, Pruning, and Disentanglement
- Multilingual Vision-Language Models, A Survey
- SuperOffload: Unleashing the Power of Large-Scale LLM Training on Superchips
- Best-of-∞ -- Asymptotic Performance of Test-Time LLM Ensembling
- A short survey on almost orthogonal vectors in a few specific large dimensions
- Selecting Open-Weight Language Models for Zero-Shot Intent Classification: A Systematic Evaluation of 41 Models
- Low-bit Model Quantization for Deep Neural Networks: A Survey
- Debiasing Multilingual LLMs in Cross-lingual Latent Space
- Speculating LLMs' Chinese Training Data Pollution from Their Tokens
- On-the-Fly Adaptation to Quantization: Configuration-Aware LoRA for Efficient Fine-Tuning of Quantized LLMs
- Towards Open-Ended Discovery for Low-Resource NLP
- LIMI: Less is More for Agency
- nDNA -- the Semantic Helix of Artificial Cognition
- CUTE: A Multilingual Dataset for Enhancing Cross-Lingual Knowledge Transfer in Low-Resource Languages
- Are you sure? Measuring models bias in content moderation through uncertainty
- Evolution of Concepts in Language Model Pre-Training
- Less Is More? Examining Fairness in Pruned Large Language Models for Summarising Opinions
- Pico: A Modular Framework for Hypothesis-Driven Small Language Model Research
- REFER: Mitigating Bias in Opinion Summarisation via Frequency Framed Prompting
- Pointing to a Llama and Call it a Camel: On the Sycophancy of Multimodal Large Language Models
- TextMineX: Data, Evaluation Framework and Ontology-guided LLM Pipeline for Humanitarian Mine Action
- Hala Technical Report: Building Arabic-Centric Instruction & Translation Models at Scale
- Pun Unintended: LLMs and the Illusion of Humor Understanding
- Differentially-private text generation degrades output language quality
- From Parameters to Performance: A Data-Driven Study on LLM Structure and Development
- MCBP: A Memory-Compute Efficient LLM Inference Accelerator Leveraging Bit-Slice-enabled Sparsity and Repetitiveness
- TigerCoder: A Novel Suite of LLMs for Code Generation in Bangla
- Building High-Quality Datasets for Portuguese LLMs: From Common Crawl Snapshots to Industrial-Grade Corpora
- QFrCoLA: a Quebec-French Corpus of Linguistic Acceptability Judgments
- From Scarcity to Efficiency: Investigating the Effects of Data Augmentation on African Machine Translation
- Llama-GENBA-10B: A Trilingual Large Language Model for German, English and Bavarian
- Crosscoding Through Time: Tracking Emergence & Consolidation Of Linguistic Representations Throughout LLM Pretraining
- FlashRecovery: Fast and Low-Cost Recovery from Failures for Large-Scale Training of LLMs
- Should LLMs be WEIRD? Exploring WEIRDness and Human Rights in Large Language Models
- Discrete Noise Inversion for Next-scale Autoregressive Text-based Image Editing
- XLQA: A Benchmark for Locale-Aware Multilingual Open-Domain Question Answering
- EPIC: Generative AI Platform for Accelerating HPC Operational Data Analytics
- APT-LLM: Exploiting Arbitrary-Precision Tensor Core Computing for LLM Acceleration
- Let's Use ChatGPT To Write Our Paper! Benchmarking LLMs To Write the Introduction of a Research Paper
- When Alignment Hurts: Decoupling Representational Spaces in Multilingual Models
- When Punctuation Matters: A Large-Scale Comparison of Prompt Robustness Methods for LLMs
- DistFlow: A Fully Distributed RL Framework for Scalable and Efficient LLM Post-Training
- More Is Better: A MoE-Based Emotion Recognition Framework with Human Preference Alignment
- FairLangProc: A Python package for fairness in NLP
- How Does Controllability Emerge In Language Models During Pretraining?
- MeshLLM: Empowering Large Language Models to Progressively Understand and Generate 3D Mesh
- LENS: Learning Ensemble Confidence from Neural States for Multi-LLM Answer Integration
- IFEvalCode: Controlled Code Generation
- Do Large Language Models Understand Morality Across Cultures?
- The Impact of Fine-tuning Large Language Models on Automated Program Repair
- Towards Inclusive NLP: Assessing Compressed Multilingual Transformers across Diverse Language Benchmarks
- SLoW: Select Low-frequency Words! Automatic Dictionary Selection for Translation on Large Language Models
- Cloud Native System for LLM Inference Serving
- Dutch CrowS-Pairs: Adapting a Challenge Dataset for Measuring Social Biases in Language Models for Dutch
- How Good LLM-Generated Password Policies Are?
- Past-Future Scheduler for LLM Serving under SLA Guarantees
- DeepSeek: Paradigm Shifts and Technical Evolution in Large AI Models
- Multilingual Multimodal Software Developer for Code Generation
- A comprehensive study of LLM-based argument classification: from LLAMA through GPT-4o to Deepseek-R1
- Beyond N-Grams: Rethinking Evaluation Metrics and Strategies for Multilingual Abstractive Summarization
- Evaluating Morphological Alignment of Tokenizers in 70 Languages
- Lost in Localization: Building RabakBench with Human-in-the-Loop Validation to Measure Multilingual Safety Gaps
- DoPI: Doctor-like Proactive Interrogation LLM for Traditional Chinese Medicine
- Advancing Financial Engineering with Foundation Models: Progress, Applications, and Challenges
- No Language Data Left Behind: A Comparative Study of CJK Language Datasets in the Hugging Face Ecosystem
- Graph Neural Networks as a Substitute for Transformers in Single-Cell Transcriptomics
- IMPACT: Inflectional Morphology Probes Across Complex Typologies
- MegaFold: Efficient Training of Next-Generation 3D Attention Protein Models on Cross-Platform GPUs
- Is There a Case for Conversation Optimized Tokenizers in Large Language Models?
- Beyond the Sentence: A Survey on Context-Aware Machine Translation with Large Language Models
- Instructing Large Language Models for Low-Resource Languages: A Systematic Study for Basque
- All is Not Lost: LLM Recovery without Checkpoints
- Assessing the Role of Data Quality in Training Bilingual Language Models
- Exploring Cultural Variations in Moral Judgments with Large Language Models
- Reviewriter: AI-Generated Instructions For Peer Review Writing
- MELABenchv1: Benchmarking Large Language Models against Smaller Fine-Tuned Models for Low-Resource Maltese NLP
- Overcoming Data Scarcity in Generative Language Modelling for Low-Resource Languages: A Systematic Review
- TokAlign: Efficient Vocabulary Adaptation via Token Alignment
- A Pre-trained Framework for Multilingual Brain Decoding Using Non-invasive Recordings
- Minimal Pair-Based Evaluation of Code-Switching
- The State of Large Language Models for African Languages: Progress and Challenges
- CC-Tuning: A Cross-Lingual Connection Mechanism for Improving Joint Multilingual Supervised Fine-Tuning
- Multilinguality Does not Make Sense: Investigating Factors Behind Zero-Shot Transfer in Sense-Aware Tasks
- Emergent Abilities of Large Language Models under Continued Pretraining for Language Adaptation
- EmotionTalk: An Interactive Chinese Multimodal Emotion Dataset With Rich Annotations
- FAMA: The First Large-Scale Open-Science Speech Foundation Model for English and Italian
- New Tools are Needed for Tracking Adherence to AI Model Behavioral Use Clauses
- DES-LOC: Desynced Low Communication Adaptive Optimizers for Training Foundation Models
- Pretraining Language Models to Ponder in Continuous Space
- DetailFlow: 1D Coarse-to-Fine Autoregressive Image Generation via Next-Detail Prediction
- Test-Time Learning for Large Language Models
- Adversarial Attacks against Closed-Source MLLMs via Feature Optimal Alignment
- Explaining Large Language Models with gSMILE
- Pretrained LLMs Learn Multiple Types of Uncertainty
- Optimization-Inspired Few-Shot Adaptation for Large Language Models
- Breaking mBad! Supervised Fine-tuning for Cross-Lingual Detoxification
- SSR-Zero: Simple Self-Rewarding Reinforcement Learning for Machine Translation
- Semantic Pivots Enable Cross-Lingual Transfer in Large Language Models
- BanglaByT5: Byte-Level Modelling for Bangla
- HausaNLP: Current Status, Challenges and Future Directions for Hausa Natural Language Processing
- FuxiMT: Sparsifying Large Language Models for Chinese-Centric Multilingual Machine Translation
- Unraveling Interwoven Roles of Large Language Models in Authorship Privacy: Obfuscation, Mimicking, and Verification
- MR. Judge: Multimodal Reasoner as a Judge
- HBO: Hierarchical Balancing Optimization for Fine-Tuning Large Language Models
- Semantic Aware Linear Transfer by Recycling Pre-trained Language Models for Cross-lingual Transfer
- Task-Core Memory Management and Consolidation for Long-term Continual Learning
- A Survey on Large Language Models in Multimodal Recommender Systems
- Lossless Compression for LLM Tensor Incremental Snapshots
- Bridging the English-Arabic Medical Knowledge Gap: Targeted Low-Rank Adaptation via Causal Layer Selection
- Large Language Models for Computer-Aided Design: A Survey
- From OSS to Open Source AI: an Exploratory Study of Collaborative Development Paradigm Divergence
- Exploring the Feasibility of Multilingual Grammatical Error Correction with a Single LLM up to 9B parameters: A Comparative Study of 17 Models
- Scalable Multi-Stage Influence Function for Large Language Models via Eigenvalue-Corrected Kronecker-Factored Parameterization
- Enhancing LLM Code Generation: A Systematic Evaluation of Multi-Agent Collaboration and Runtime Debugging for Improved Accuracy, Reliability, and Latency
- Learning a Vector-Symbolic Model for Socio-Cultural Tasks
- AdCare-VLM: Towards a Unified and Pre-aligned Latent Representation for Healthcare Video Understanding
- Using large language models to extract plant functional traits from unstructured text
- How Private Are DNA Embeddings? Inverting Foundation Model Representations of Genomic Sequences
- A comprehensive study of LLM-based argument classification: from Llama through DeepSeek to GPT-5.2
- Progressive Training for Explainable Citation-Grounded Dialogue: Reducing Hallucination to Zero in English-Hindi LLMs
- Precision Where It Matters: A Novel Spike Aware Mixed-Precision Quantization Strategy for LLaMA-based Language Models
- Multimodal Large Language Models for Medicine: A Comprehensive Survey
- Enhancing LLM Language Adaption through Cross-lingual In-Context Pre-training
- Enhancing Non-Core Language Instruction-Following in Speech LLMs via Semi-Implicit Cross-Lingual CoT Reasoning
- Inside the LLM Word Factory
- Transformer Scalability Crisis: The First Comprehensive Empirical Analysis of Performance Walls in Modern Language Models
- MER 2025: When Affective Computing Meets Large Language Models
- Adaptra: Straggler-Resilient Hybrid-Parallel Training with Pipeline Adaptation
- AndroidGen: Building an Android Language Agent under Data Scarcity
- SolarGPT-QA: A Domain-Adaptive Large Language Model for Educational Question Answering in Space Weather and Heliophysics
- Attention Needs to Focus: A Unified Perspective on Attention Allocation
- How Effective are Generative Large Language Models in Performing Requirements Classification?
- Lost in Multilinguality: Dissecting Cross-lingual Factual Inconsistency in Transformer Language Models
- Optimizing LLMs for Italian: Reducing Token Fertility and Enhancing Efficiency Through Vocabulary Adaptation
- A Survey of Foundation Model-Powered Recommender Systems: From Feature-Based, Generative to Agentic Paradigms
- Saliency-driven Dynamic Token Pruning for Large Language Models
- The Bitter Lesson Learned from 2,000+ Multilingual Benchmarks
- SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech Translation
- Kuwain 1.5B: An Arabic SLM via Language Injection
- Hardware-based Heterogeneous Memory Management for Large Language Model Inference
- Analyzing LLMs' Knowledge Boundary Cognition Across Languages Through the Lens of Internal Representations
- Analysing the Robustness of Vision-Language-Models to Common Corruptions
- VLMGuard-R1: Proactive Safety Alignment for VLMs via Reasoning-Driven Prompt Optimization
- Tilus: A Tile-Level GPGPU Programming Language for Low-Precision Computation
- Learning to Be A Doctor: Searching for Effective Medical Agent Architectures
- Training LLMs on HPC Systems: Best Practices from the OpenGPT-X Project
- A Survey of Reasoning with Foundation Models: Concepts, Methodologies, and Outlook
- Redefining Machine Translation on Social Network Services with Large Language Models
- SEA-LION: Southeast Asian Languages in One Network
- Llama-3-Nanda-10B-Chat: An Open Generative Large Language Model for Hindi
- Stock Market Forecasting: From Traditional Predictive Models to Large Language Models
- Learning Natural Language Constraints for Safe Reinforcement Learning of Language Agents
- LLM for Complex Reasoning Task: An Exploratory Study in Fermi Problems
Discussions
Related