Parameterized Synthetic Text Generation with SimpleStories

2025/04/12 by Lennart Finke, Chandan Sreedhara, Finke, Lennart +18 · 1 voice · 1 citation
Computer Science · #Natural Language Processing Techniques #Semantic Web and Ontologies #Topic Modeling #cs.AI #cs.CL

paper · pdf · doi:10.48550/arxiv.2504.09184

openalex publication_date 2025/04/12 · openalex created_date 2025/10/14 · openalex updated_date 2026/07/28

Abstract

We present SimpleStories, a large synthetic story dataset in simple language, consisting of 2 million samples each in English and Japanese. Through parameterizing prompts at multiple levels of abstraction, we achieve control over story characteristics at scale, inducing syntactic and semantic diversity. Ablations on a newly trained model suite show improved sample efficiency and model interpretability compared to the TinyStories dataset. We open-source all constituent parts of model creation, hoping to enable novel ways to study the end-to-end training process. As a byproduct, we move the frontier regarding the fewest-parameter language model that outputs grammatical natural language.

Cited by

• Spectral Reach: Understanding Neural Scaling as Progress into the Spectral Tail

Discussions

• 📄 Paper: arxiv.org/abs/2504.09184 ✨ Interactive Viz: fi-le.net/simplestories 📚 Datasets + Model Suite: huggingface.co/SimpleStories ⚙️ Generation Code: github.com/lennart-fink... 🧠 Training Code: g [bsky, 6 points, 0 comments]

Related

• BERTtime Stories: Investigating the Role of Synthetic Story Data in Language Pre-training
• TinyStories: How Small Can Language Models Be and Still Speak Coherent English?
• Generating Datasets with Pretrained Language Models
• Automatic and Human-AI Interactive Text Generation
• A Plug-and-Play Method for Controlled Text Generation