Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models

2023/12/11 by Avi Singh, Singh, Avi, John D. Co-Reyes +76 · 37 citations
Computer Science · #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning and Data Classification #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2312.06585

openalex publication_date 2023/12/11 · openalex created_date 2023/12/13 · openalex updated_date 2026/07/28

Abstract

Fine-tuning language models~(LMs) on human-generated data remains a prevalent practice. However, the performance of such models is often limited by the quantity and diversity of high-quality human data. In this paper, we explore whether we can go beyond human data on tasks where we have access to scalar feedback, for example, on math problems where one can verify correctness. To do so, we investigate a simple self-training method based on expectation-maximization, which we call ReST^EM, where we (1) generate samples from the model and filter them using binary feedback, (2) fine-tune the model on these samples, and (3) repeat this process a few times. Testing on advanced MATH reasoning and APPS coding benchmarks using PaLM-2 models, we find that ReST^EM scales favorably with model size and significantly surpasses fine-tuning only on human data. Overall, our findings suggest self-training with feedback can substantially reduce dependence on human-generated data.

Cited by

• From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement
• Learning to Reason in LLMs by Expectation Maximization
• VulnLLM-R: Specialized Reasoning LLM with Agent Scaffold for Vulnerability Detection
• WebSTAR: Scalable Data Synthesis for Computer Use Agents with Step-Level Filtering
• From Solving to Verifying: A Unified Objective for Robust Reasoning in LLMs
• ACT as Human: Multimodal Large Language Model Data Annotation with Critical Thinking
• Through the Judge's Eyes: Inferred Thinking Traces Improve Reliability of LLM Raters
• Teaching Models to Teach Themselves: Reasoning at the Edge of Learnability
• Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning
• PlanU: Large Language Model Reasoning through Planning under Uncertainty
• EvoSyn: Generalizable Evolutionary Data Synthesis for Verifiable Learning
• Confidence as a Reward: Transforming LLMs into Reward Models
• Putting on the Thinking Hats: A Survey on Chain of Thought Fine-tuning from the Perspective of Human Reasoning Mechanism
• HoneyBee: Data Recipes for Vision-Language Reasoners
• Learning a Dense Reasoning Reward Model from Expert Demonstration via Inverse Reinforcement Learning
• Reinforce-Ada: An Adaptive Sampling Framework under Non-linear RL Objectives
• GuidedSampling: Steering LLMs Towards Diverse Candidate Solutions at Inference-Time
• GEM: A Gym for Agentic LLMs
• Diffusion Alignment as Variational Expectation-Maximization
• GRPO-λ: Credit Assignment improves LLM Reasoning
• SIRI: Scaling Iterative Reinforcement Learning with Interleaved Compression
• AceSearcher: Bootstrapping Reasoning and Search for LLMs via Reinforced Self-Play
• Think Right, Not More: Test-Time Scaling for Numerical Claim Verification
• LOCA: Logical Chain Augmentation for Scientific Corpus Cleaning
• Exploiting Tree Structure for Credit Assignment in RL Training of LLMs
• GPO: Learning from Critical Steps to Improve LLM Reasoning
• Leveraging Large Language Models to Effectively Generate Visual Data for Canine Musculoskeletal Diagnoses
• Bridging the Capability Gap: Joint Alignment Tuning for Harmonizing LLM-based Multi-Agent Systems
• Learn from What We HAVE: History-Aware VErifier that Reasons about Past Interactions Online
• ReST-RL: Achieving Accurate Code Reasoning of LLMs with Optimized Self-Training and Decoding
• Utilizing Training Data to Improve LLM Reasoning for Tabular Understanding
• SYNAPSE-G: Bridging Large Language Models and Graph Learning for Rare Event Classification
• Modelling and Classifying the Components of a Literature Review
• Self-Questioning Language Models
• Lessons from complex systems science for AI governance
• CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning tasks
• PITA: Preference-Guided Inference-Time Alignment for LLM Post-Training

Related

• Think, Prune, Train, Improve: Scaling Reasoning without Scaling Models
• Can Large Language Models Invent Algorithms to Improve Themselves?: Algorithm Discovery for Recursive Self-Improvement through Reinforcement Learning
• Teaching Language Models to Self-Improve through Interactive Demonstrations
• Post-Training Large Language Models via Reinforcement Learning from Self-Feedback
• Large Language Models Can Self-Improve