LLMs Can Teach Themselves to Better Predict the Future

2025/02/07 by Benjamin Turtel, Danny Franklin, Turtel, Benjamin +3 · 10 voices · 3 citations
Computer Science · Psychology · #Psychology #Research Data Management Practices #cs.AI #cs.CL

paper · pdf · doi:10.48550/arxiv.2502.05253

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2025/02/07 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We present an outcome-driven fine-tuning framework that enhances the forecasting capabilities of large language models (LLMs) without relying on human-curated reasoning samples. Our method leverages model self-play to generate pairs of diverse reasoning trajectories and probabilistic forecasts for a set of diverse questions that resolve after the models' knowledge cutoff date. We then rank pairs of these reasoning traces by their distance to the actual outcomes before fine-tuning the model via Direct Preference Optimization (DPO). On a separate test set, our approach increases prediction accuracy of Phi-4 14B and DeepSeek-R1 14B by between 7--10% over a base model and a DPO fine-tuned control model with randomized labels, bringing them on par with forecasting capabilities of much larger frontier models like GPT-4o.

Citations

• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models
• Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm

Cited by

• Outcome-based Reinforcement Learning to Predict the Future
• AIA Forecaster: Technical Report
• Advancing Event Forecasting through Massive Training of Large Language Models: Challenges, Solutions, and Broader Impacts

Discussions

• LLMs can teach themselves to better predict the future [hn, 176 points, 86 comments]
• Yay, AI can teach itself to be a better prediction machine. Affording myself some license, that's basically saying a brain can teach itself to be a better brain. Our brains are prediction machines for [bsky, 1 points, 1 comments]
• https://arxiv.org/abs/2502.05253 #cs.CL #cs.AI Event Attributes [bsky, 0 points, 0 comments]
• LLMs Can Teach Themselves to Better Predict the Future [bsky, 0 points, 0 comments]
• LLMs Can Teach Themselves to Better Predict the Future #HackerNews arxiv.org/abs/... [bsky, 0 points, 0 comments]
• LLMs Can Teach Themselves to Better Predict the Future https://arxiv.org/abs/2502.05253 (https://news.ycombinator.com/item?id=43014918) [bsky, 0 points, 0 comments]
• LLMs can teach themselves to better predict the future https://arxiv.org/abs/2502.05253 [comments] [113 points] [bsky, 0 points, 0 comments]
• LLMs Can Teach Themselves to Better Predict the Future (arxiv.org) Main Link | Discussion [bsky, 0 points, 0 comments]
• LLMs can teach themselves to better predict the future https://arxiv.org/abs/2502.05253 https://news.ycombinator.com/item?id=43014918 [bsky, 0 points, 0 comments]
• LLMs Can Teach Themselves to Better Predict the Future https://arxiv.org/abs/2502.05253 (https://news.ycombinator.com/item?id=43014918) [bsky, 0 points, 0 comments]

Related

• Can Language Models Use Forecasting Strategies?
• Evaluating LLMs on Real-World Forecasting Against Expert Forecasters
• Beyond Naïve Prompting: Strategies for Improved Zero-shot Context-aided Forecasting with LLMs
• Future Is Unevenly Distributed: Forecasting Ability of LLMs Depends on What We're Asking
• Approaching Human-Level Forecasting with Language Models