Textbooks Are All You Need

2023/06/20 by Suriya Gunasekar, Yi Zhang, Gunasekar, Suriya +35 · 11 voices · 55 citations
Computer Science · #Topic Modeling #Natural Language Processing Techniques #Text Readability and Simplification

paper · pdf · doi:10.48550/arxiv.2306.11644

Abstract

We introduce phi-1, a new large language model for code, with significantly smaller size than competing models: phi-1 is a Transformer-based model with 1.3B parameters, trained for 4 days on 8 A100s, using a selection of ``textbook quality" data from the web (6B tokens) and synthetically generated textbooks and exercises with GPT-3.5 (1B tokens). Despite this small scale, phi-1 attains pass@1 accuracy 50.6% on HumanEval and 55.5% on MBPP. It also displays surprising emergent properties compared to phi-1-base, our model before our finetuning stage on a dataset of coding exercises, and phi-1-small, a smaller model with 350M parameters trained with the same pipeline as phi-1 that still achieves 45% on HumanEval.

Cited by

• OntoBook: Ontology-Grounded Synthetic Textbooks for Medical Encoder Pretraining
• Spaghetti Architect: A Contamination-Resistant, By-Construction-Labelled, Multi-Language Code Dataset Generator
• RF-Agent: A Practical Framework for Building Language Agents for RFIC Design
• FinRAG-12B: A Production-Validated Recipe for Grounded Question Answering in Banking
• Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergence
• DataPrep-Bench: Benchmarking LLMs as Training Data Preparators
• GLAN-QnA-KR: A Seedless Taxonomy-Driven Korean Instruction Corpus
• Self-supervised learning of molecular representations from millions of tandem mass spectra using DreaMS
• Smaller But Better: Unifying Layout Generation with Smaller Large Language Models
• Position: It's Time to Act on the Risk of Efficient Personalized Text Generation
• Beyond Shapley: An Influence-Based Data Auditing Pipeline for LLM Alignment and Evaluation
• MAction-SocialNav: Multi-Action Socially Compliant Navigation via Reasoning-enhanced Prompt Tuning
• Effect of Document Packing on the Latent Multi-Hop Reasoning Capabilities of Large Language Models
• GLaD: Geometric Latent Distillation for Vision-Language-Action Models
• Persian-Phi: Efficient Cross-Lingual Adaptation of Compact LLMs via Curriculum Learning
• CryptoQA: A Large-scale Question-answering Dataset for AI-assisted Cryptography
• Video-R2: Reinforcing Consistent and Grounded Reasoning in Multimodal Language Models
• Geometry of Decision Making in Language Models
• Online Data Curation for Object Detection via Marginal Contributions to Dataset-level Average Precision
• On the Notion that Language Models Reason
• Sensitivity of Small Language Models to Fine-tuning Data Contamination
• OMPILOT: Harnessing Transformer Models for Auto Parallelization to Shared Memory Computing Paradigms
• A CPU-Centric Perspective on Agentic AI
• Validity Is What You Need
• PairUni: Pairwise Training for Unified Multimodal Language Models
• OrgForge: A Multi-Agent Simulation Framework for Verifiable Synthetic Corporate Corpora
• SENSE: Efficient EEG-to-Text via Privacy-Preserving Semantic Retrieval
• Beyond Length: Quantifying Long-Range Information for Long-Context LLM Pretraining Data
• Large Language Model for Verilog Code Generation: Literature Review and the Road Ahead
• Flight Delay Prediction via Cross-Modality Adaptation of Large Language Models and Aircraft Trajectory Representation
• Chain of Execution Supervision Promotes General Reasoning in Large Language Models
• Data-Centric Lessons To Improve Speech-Language Pretraining
• Learning from the Best, Differently: A Diversity-Driven Rethinking on Data Selection
• Reasoning Distillation and Structural Alignment for Improved Code Generation
• Synthesizing Agentic Data for Web Agents with Progressive Difficulty Enhancement Mechanisms
• Iterative Data Curation with Theoretical Guarantees
• KORMo: Korean Open Reasoning Model for Everyone
• AI-generated text boundary detection with RoFT
• Webscale-RL: Automated Data Pipeline for Scaling RL Data to Pretraining Levels
• Uncovering the Computational Ingredients of Human-Like Representations in LLMs
• Characterizing Model Behavior Under Synthetic Data Training: An Empirical Study Across Scales and Mixing Ratios
• Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignment
• From silicon to solutions: AI's impending impact on research and discovery
• PseudoBridge: Pseudo Code as the Bridge for Better Semantic and Logic Alignment in Code Retrieval
• Thinking Augmented Pre-training
• Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Training
• How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data
• Reading the unreadable: creating a dataset of 19th century English newspapers using image-to-text language models
• Generating High-Quality Datasets for Code Editing via Open-Source Language Models
• Enhancing Generalization in Vision-Language-Action Models by Preserving Pretrained Representations
• SciGPT: A Large Language Model for Scientific Literature Understanding and Knowledge Discovery
• Interface on demand: Towards AI native Control interfaces for 6G
• Learning Facts at Scale with Active Reading
• LLM Unlearning Without an Expert Curated Dataset
• List of large language models [wikipedia]

Discussions

• Textbooks are all you need [hn, 256 points, 106 comments]
• Textbooks Are All You Need [lemmy, 7 points, 0 comments]
• Textbooks Are All You Need. 1.3B LLM trained on 51B tokens hits 51% on HumanEval. (20.06.2023 paper) [lemmy, 5 points, 1 comments]
• On the quality-over-quantity point specifically: Textbooks Are All You Need, arxiv.org/abs/2306.11644. And if data-constrained becomes the real regime: arxiv.org/abs/2305.16264. [bsky, 3 points, 1 comments]
• AI Is Already Crushing the News Industry [lemmy, 3 points, 0 comments]
• via Tim Hanson: https://arxiv.org/abs/2306.11644 Textbooks are all you need Demonstrates that by training on a high-quality yet small dataset (7B tokens, 7 epochs), a small network (1.3B parameters) [bsky, 0 points, 1 comments]
• I'd not seen this, using "synthetically generated textbooks and exercises" arxiv.org/abs/2306.11644 [bsky, 0 points, 0 comments]
• 4. GPT 처음 튀어 나왔을때 규모를 듣고 깜짝 놀랬는데 LLAMA부터 시작해서 (7B나 13B긴 하지만) 로컬 GPU에서도 뭔가 돌릴 수 있게 됐잖아요? 또 https://arxiv.org/abs/2306.11644 이런 친구도 있구요 소프트웨어적으로 개선의 여지가 있는 거 아닐까? 하는 생각도 들구요 [bsky, 0 points, 1 comments]
• phi-1: a 1.3B code generation model comparable to SoTA models such as WizardCoder or GPTs. This looks great, although it has some tricks. Also, I guess it would work even on the Raspberry Pi. https:/ [bsky, 0 points, 1 comments]
• What's you doing is not a compression. It's a topology. Microsoft made almost similar with their Phi LM family. They limited the knowledge base for the first stage and then taught the model on the wid [bsky, 0 points, 0 comments]
• Here's one source: arxiv.org/abs/2306.11644 [bsky, 0 points, 1 comments]

Related

• Textbooks Are All You Need II: phi-1.5 technical report
• Phi-4 Technical Report
• MHPP: Exploring the Capabilities and Limitations of Language Models Beyond Basic Code Generation
• Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone
• Pretraining on the Test Set Is All You Need