ProgramBench: Can Language Models Rebuild Programs From Scratch?
2026/05/05 by John Yang, Kilian Lieret, Jeffrey Ma +9 · 17 voices · 6 citations
#cs.SE #cs.AI
paper · pdf
Abstract
Turning ideas into full software projects from scratch has become a popular use case for language models. Agents are being deployed to seed, maintain, and grow codebases over extended periods with minimal human oversight. Such settings require models to make high-level software architecture decisions. However, existing benchmarks measure focused, limited tasks such as fixing a single bug or developing a single, specified feature. We therefore introduce ProgramBench to measure the ability of software engineering agents to develop software holisitically. In ProgramBench, given only a program and its documentation, agents must architect and implement a codebase that matches the reference executable's behavior. End-to-end behavioral tests are generated via agent-driven fuzzing, enabling evaluation without prescribing implementation structure. Our 200 tasks range from compact CLI tools to widely used software such as FFmpeg, SQLite, and the PHP interpreter. We evaluate 9 LMs and find that none fully resolve any task, with the best model passing 95% of tests on only 3% of tasks. Models favor monolithic, single-file implementations that diverge sharply from human-written code.
Citations
Cited by
Discussions
- ProgramBench: Can language models rebuild programs from scratch? [hn, 150 points, 80 comments]
- Apparently, the answer is: No. arxiv.org/abs/2605.03546 [bsky, 30 points, 4 comments]
- ProgramBench, a new benchmark from Facebook/Meta(by SWE-Bench creators) to see if LLMs can recreate real executable programs (ffmpeg, SQLite) from scratch with no internet access- They all score 0%. [lemmy, 6 points, 1 comments]
- Cheating for the win arxiv.org/pdf/2605.03546 [bsky, 1 points, 0 comments]
- Esta interesante este paper, desarrollaron un benchmark que pide a múltiples LLM reconstruir software crítico (ffmpeg, SQLite, un intérprete PHP) sin internet y desde cero. Ninguno logra pasar testing [bsky, 1 points, 0 comments]
- ProgramBench: Can language models rebuild programs from scratch? https://arxiv.org/abs/2605.03546 https://news.ycombinator.com/item?id=48045174 [bsky, 0 points, 0 comments]
- ProgramBench: Can Language Models Rebuild Programs from Scratch? https://arxiv.org/abs/2605.03546 [bsky, 0 points, 0 comments]
- 📰 ProgramBench: Can language models rebuild programs from scratch? 🔗 https://arxiv.org/abs/2605.03546 💬 Discuss on HN [bsky, 0 points, 0 comments]
- The paper: arxiv.org/abs/2605.03546 [bsky, 0 points, 1 comments]
- Paper: arxiv.org/abs/2605.03546 Website: programbench.com [bsky, 0 points, 1 comments]
- META is launching a new benchmark that aims to test LLMs on building large-scale / end-to-end applications like ffmpeg / sqlite / interpreters from docs, instead of just snippets/pull-requests: arxiv. [bsky, 0 points, 1 comments]
- Links: abs, pdf Search: Bluesky, Twitter, Reddit, Hacker News, Hugging Face, alphaXiv [bsky, 0 points, 1 comments]
- 📰 ProgramBench, an AI project, aims to rebuild programs from scratch using language models, potentially revolutionizing software development and programming education. 🔗 https://arxiv.org/abs/2605.0 [bsky, 0 points, 0 comments]
- ProgramBench: Can language models rebuild programs from scratch? https:// arxiv.org/abs/2605.03546 # arxiv [mastodon, 0 points, 0 comments]
- ProgramBench: Can Language Models Rebuild Programs from Scratch? https://arxiv.org/abs/2605.03546 (https://news.ycombinator.com/item?id=48045174) [bsky, 0 points, 0 comments]
- ProgramBench: Can Language Models Rebuild Programs from Scratch? https://arxiv.org/abs/2605.03546 (https://news.ycombinator.com/item?id=48045174) [bsky, 0 points, 0 comments]
- ProgramBench: Can Language Models Rebuild Programs from Scratch? [bsky, 0 points, 0 comments]
Related