TinyStories: How Small Can Language Models Be and Still Speak Coherent English?

2023/05/12 by Ronen Eldan, Yuanzhi Li, Eldan, Ronen +1 · 19 voices · 77 citations
Computer Science · #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling #cs.AI #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.2305.07759

openalex publication_date 2023/05/12 · openalex created_date 2023/05/17 · openalex updated_date 2026/07/28

Abstract

Language models (LMs) are powerful tools for natural language processing, but they often struggle to produce coherent and fluent text when they are small. Models with around 125M parameters such as GPT-Neo (small) or GPT-2 (small) can rarely generate coherent and consistent English text beyond a few words even after extensive training. This raises the question of whether the emergence of the ability to produce coherent English text only occurs at larger scales (with hundreds of millions of parameters or more) and complex architectures (with many layers of global attention). In this work, we introduce TinyStories, a synthetic dataset of short stories that only contain words that a typical 3 to 4-year-olds usually understand, generated by GPT-3.5 and GPT-4. We show that TinyStories can be used to train and evaluate LMs that are much smaller than the state-of-the-art models (below 10 million total parameters), or have much simpler architectures (with only one transformer block), yet still produce fluent and consistent stories with several paragraphs that are diverse and have almost perfect grammar, and demonstrate reasoning capabilities. We also introduce a new paradigm for the evaluation of language models: We suggest a framework which uses GPT-4 to grade the content generated by these models as if those were stories written by students and graded by a (human) teacher. This new paradigm overcomes the flaws of standard benchmarks which often requires the model's output to be very structures, and moreover provides a multidimensional score for the model, providing scores for different capabilities such as grammar, creativity and consistency. We hope that TinyStories can facilitate the development, analysis and research of LMs, especially for low-resource or specialized domains, and shed light on the emergence of language capabilities in LMs.

Cited by

• A Structural Interpretation of GELU and Threshold-Transmission Activations via the First-Order Loss Function
• Pretraining Recurrent Networks without Recurrence
• Per-Token Fixed-Point Convergence in Depth-Recurrent Transformers
• SR-TTT Does Not Learn Retrieval: A Correction and Mechanistic Post-Mortem of Surprisal-Aware Residual Test-Time Training
• Moving Alphabet: A Controlled Study of Training Data for Text-to-Video Generation
• Can Transformers Really Do It All? On the Compatibility of Inductive Biases Across Tasks
• A Controlled Study of Attention-Only Transformers
• Kolmogorov--Arnold Networks for Small Language Models
• Explaining Attention with Program Synthesis
• Language Models are Injective and Hence Invertible
• Every Language Model Has a Forgery-Resistant Signature
• Self-Adapting Language Models
• Leaner Transformers: More Heads, Less Depth
• Towards Robust Reinforcement Learning for Small-Scale Language Model Agents
• Through the Bottleneck: How Multi-head Latent Attention Separates Content from Position in Language Models
• GQ-VAE: A gated quantized VAE for learning variable length tokens
• SMART SLM: Structured Memory and Reasoning Transformer, A Small Language Model for Accurate Document Assistance
• Ky Fan Norms and Beyond: Dual Norms and Combinations for Matrix Optimization
• Persian-Phi: Efficient Cross-Lingual Adaptation of Compact LLMs via Curriculum Learning
• Generating Storytelling Images with Rich Chains-of-Reasoning
• Beyond Data Filtering: Knowledge Localization for Capability Removal in LLMs
• TaleFrame: An Interactive Story Generation System with Fine-Grained Control and Large Language Models
• Cross-Lingual Interleaving for Speech Language Models
• Length-MAX Tokenizer for Language Models
• Weight-sparse transformers have interpretable circuits
• Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Data
• On the Notion that Language Models Reason
• Synthetic Data-Driven Prompt Tuning for Financial QA over Tables and Documents
• Next-Latent Prediction Transformers Learn Compact World Models
• S²LM: Towards Semantic Steganography via Large Language Models
• In Good GRACEs: Principled Teacher Selection for Knowledge Distillation
• Sample-Efficient Language Modeling with Linear Attention and Lightweight Enhancements
• Gaperon: A Peppered English-French Generative Language Model Suite
• Reactive Transformer (RxT) -- Stateful Real-Time Processing for Event-Driven Reactive Language Models
• BitSkip: An Empirical Analysis of Quantization and Early Exit Composition
• How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Data
• Blackbox Model Provenance via Palimpsestic Membership Inference
• Closing the Gap Between Text and Speech Understanding in LLMs
• CurLL: A Developmental Framework to Evaluate Continual Learning in Language Models
• QLENS: Towards A Quantum Perspective of Language Transformers
• Calibrating Generative Models
• Sparsely gated tiny linear experts
• Sparse Query Attention (SQA): A Computationally Efficient Attention Mechanism with Query Heads Reduction
• Guided Star-Shaped Masked Diffusion
• Pool Me Wisely: On the Effect of Pooling in Transformer-Based Models
• Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls
• How does the optimizer implicitly bias the model merging loss landscape?
• RACE Attention: A Strictly Linear-Time Attention Layer for Training on Outrageously Large Contexts
• Composer: A Search Framework for Hybrid Neural Architecture Design
• Feature Identification via the Empirical NTK
• Decomposing Attention To Find Context-Sensitive Neurons
• Data Selection for Fine-tuning Vision Language Models via Cross Modal Alignment Trajectories
• SeedPrints: Fingerprints Can Even Tell Which Seed Your Large Language Model Was Trained From
• Controlled Generation for Private Synthetic Text
• Dynamics of Learning: Generative Schedules from Latent ODEs
• Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Training
• BridgeAlign: Bridging Preference Alignment for Humanities and Social Sciences
• HSS-Synth: Humanities and Social Sciences Data Synthesis for LLMs
• Rethinking the Role of Text Complexity in Language Model Pretraining
• Pico: A Modular Framework for Hypothesis-Driven Small Language Model Research
• Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models
• SpeechWeave: Diverse Multilingual Synthetic Text & Audio Data Generation Pipeline for Training Text to Speech Models
• Building Large-Scale English-Romanian Literary Translation Resources with Open Models
• Towards Fundamental Language Models: Does Linguistic Competence Scale with Model Size?
• Preserving Bilinear Weight Spectra with a Signed and Shrunk Quadratic Activation Function
• Gated Associative Memory: A Parallel O(N) Architecture for Efficient Sequence Modeling
• Igniting Creative Writing in Small Language Models: LLM-as-a-Judge versus Multi-Agent Refined Rewards
• TinyServe: Query-Aware Cache Selection for Efficient LLM Serving
• WISCA: A Lightweight Model Transition Method to Improve LLM Training via Weight Scaling
• One Shot vs. Iterative: Rethinking Pruning Strategies for Model Compression
• BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining
• MiCo: End-to-End Mixed Precision Neural Network Co-Exploration Framework for Edge AI
• Can Large Language Models Generate Effective Datasets for Emotion Recognition in Conversations?
• SONAR-LLM: Autoregressive Transformer that Thinks in Sentence Embeddings and Speaks in Tokens
• Investigating Gender Bias in LLM-Generated Stories via Psychological Stereotypes
• PaPaformer: Language Model from Pre-trained Parallel Paths
• Embryology of a Language Model

Discussions

• TinyStories: How Small Can Language Models Be and Still Speak Coherent English? (2023) [hn, 218 points, 104 comments]
• How Small Can Language Models Be and Still Speak Coherent English? [hn, 42 points, 10 comments]
• Morning. After an unproductive day yesterday I've decided to pivot. I found a paper that explores how small a transformer can be and still speak English. A small model with a simple architecture can d [bsky, 13 points, 2 comments]
• Relevant to your interests. Influential dataset, consider this an entrypoint. Strong recommend poking around the tree of works that have cited this. Loads of interesting stuff in the minimal training [bsky, 5 points, 2 comments]
• Meanwhile LLMs largely do not struggle with grammar, even though grammar completely eludes us as a formally definable thing. We know the k-complexity of grammar must not be that high because even very [bsky, 4 points, 1 comments]
• TinyStories: How Small Can Language Models Be and Still Speak Coherent English? [lobsters, 3 points, 0 comments]
• How small can a #AI language model be? [bsky, 3 points, 0 comments]
• TinyStories (arxiv.org/abs/2305.07759) has been extremely useful for researchers, though is quite formulaic (e.g. 59% start with "Once upon a time"). We address this with parametrized prompts, allowin [bsky, 2 points, 1 comments]
• TinyStories: How Small Can Language Models Be and Still Speak Coherent English? (2023) https://arxiv.org/abs/2305.07759 https://news.ycombinator.com/item?id=42576755 [bsky, 1 points, 0 comments]
• TinyStories: How Small Can Language Models Be and Still Speak Coherent English? (arxiv.org) Main Link | Discussion [bsky, 0 points, 0 comments]
• TinyStories: How Small Can Language Models Be and Still Speak Coherent English? view on hacker news [bsky, 0 points, 0 comments]
• Have you seen arxiv.org/abs/2305.07759 [bsky, 0 points, 0 comments]
• I don’t have time to argue with a bunch of AI skeptics on the internet, but those who are actually curious and want to learn, some papers to read: SLM trained on children’s books with emergent qualit [bsky, 0 points, 1 comments]
• TinyStories: How Small Can Language Models Be and Still Speak Coherent English? https://arxiv.org/abs/2305.07759 (https://news.ycombinator.com/item?id=42576755) [bsky, 0 points, 0 comments]
• TinyStories: How Small Can Language Models Be and Still Speak Coherent English? #cohere #hackernews #news [bsky, 0 points, 0 comments]
• TinyStories: How Small Can Language Models Be and Still Speak Coherent English? [bsky, 0 points, 0 comments]
• TinyStories: How Small Can Language Models Be and Still Speak Coherent English? #HackerNews arxiv.org/abs/... [bsky, 0 points, 0 comments]
• TinyStories: How Small Can Language Models Be and Still Speak Coherent English? (2023) https://arxiv.org/abs/2305.07759 [comments] [158 points] [bsky, 0 points, 0 comments]
• TinyStories: How Small Can Language Models Be and Still Speak Coherent English? https://arxiv.org/abs/2305.07759 (https://news.ycombinator.com/item?id=42576755) [bsky, 0 points, 0 comments]

Related

• Tiny language models
• TinyHelen's First Curriculum: Training and Evaluating Tiny Language Models in a Simpler Language Environment
• Baby's CoThought: Leveraging Large Language Models for Enhanced Reasoning in Compact Models
• Readability ≠ Learnability: Rethinking the Role of Simplicity in Training Small Language Models
• It's Not Just Size That Matters: Small Language Models Are Also Few-Shot\n Learners