Benchmarks in Leipzig
2026/06/04 by Andrei Balakin, Miklós Bóna, Marie-Charlotte Brandenburg +45 · 18 voices
#math.HO #cs.AI #math.AG #math.CO #math.RT
paper · pdf
Abstract
Between April 1 and May 15, 2026, a group of 49 mathematicians compiled a dataset of research-level mathematics questions with known answers. Most of the work was done during the 3-day workshop *Benchmarks in Leipzig* with 35 participants at the Max Planck Institute for Mathematics in the Sciences in Leipzig, Germany. We present the resulting collection of 100 questions. We evaluated these questions in three stages: a single attempt by five state-of-the-art LLMs, followed by a 20-runs-per-model evaluation with three of these models, and finally a 3-run attempt with two heavy-thinking models. After Stage 1, 41 questions remained completely unsolved; after Stage 2, this count dropped to 16; and we concluded Stage 3 with only 2 unsolved questions. This demonstrates that the mathematical reasoning capabilities of LLMs are becoming impressive.
Citations
Discussions
- Benchmarks in Leipzig [hn, 138 points, 49 comments]
- feels like an album title maybe Ultravox? [bsky, 4 points, 0 comments]
- Here's what nobody's telling you: Germany just set a new benchmark in Leipzig, but it's not the Champions League score we're talking about... Read the surprising math behind "Benchmarks in Leipzig" no [bsky, 2 points, 1 comments]
- "Benchmarks in Leipzig": 49 researchers, 100 research-level mathematics questions: Only 2 of 100 questions remained unsolved. Performance shows significant variation across different models and repeat [bsky, 1 points, 0 comments]
- Benchmarks in Leipzig #HackerNews https://arxiv.org/abs/2606.05818 [bsky, 0 points, 0 comments]
- Benchmarks in Leipzig https://arxiv.org/abs/2606.05818 https://news.ycombinator.com/item?id=48425247 [bsky, 0 points, 0 comments]
- Benchmarks in Leipzig https://arxiv.org/abs/2606.05818 [bsky, 0 points, 0 comments]
- Benchmarks in Leipzig https://arxiv.org/abs/2606.05818 [comments] [39 points] [bsky, 0 points, 0 comments]
- Benchmarks in Leipzig https://arxiv.org/abs/2606.05818 (https://news.ycombinator.com/item?id=48425247) [bsky, 0 points, 0 comments]
- Andrei Balakin, et al.: Benchmarks in Leipzig https://arxiv.org/abs/2606.05818 https://arxiv.org/pdf/2606.05818 https://arxiv.org/html/2606.05818 [bsky, 0 points, 0 comments]
- 📰 Benchmarks in Leipzig: A study published on arXiv.org and discussed on Hacker News highlights critical constraints for professional news reporting. 🔗 https://arxiv.org/abs/2606.05818 #Tech #Dev [bsky, 0 points, 0 comments]
- Benchmarks in Leipzig view on hacker news [bsky, 0 points, 0 comments]
- Benchmarks in Leipzig https:// arxiv.org/abs/2606.05818 # arxiv [mastodon, 0 points, 0 comments]
- Benchmarks in Leipzig https:// arxiv.org/abs/2606.05818 # HackerNews # Benchmarks # Leipzig # research # arxiv # technology # innovation [mastodon, 0 points, 0 comments]
- 📰 Benchmarks in Leipzig 🔗 https://arxiv.org/abs/2606.05818 💬 Discuss on HN [bsky, 0 points, 0 comments]
- Benchmarks in Leipzig https://arxiv.org/abs/2606.05818 (https://news.ycombinator.com/item?id=48425247) [bsky, 0 points, 0 comments]
- https://bsky.app/profile/hackernews.com.web.brid.gy/post/3mnn4cpz5lja2 [bsky, 0 points, 0 comments]
- Benchmarks in Leipzig [bsky, 0 points, 0 comments]
Related