EsoLang-Bench: Evaluating Genuine Reasoning in Large Language Models via Esoteric Programming Languages

2026/03/10 by Aman Sharma, Paras Chopra · 10 voices
#cs.AI #cs.LG #cs.SE

paper · pdf

Abstract

Large language models achieve near-ceiling performance on code generation benchmarks, yet most of the programming languages used by popular benchmarks such as SWE-bench and HumanEval (e.g. Python, JavaScript) are squarely in-distribution. They appear at scale in pre-training corpora and are heavily reinforced during post-training. To study LLM performance on unfamiliar programming languages, we introduce EsoLang-Bench, a benchmark using five esoteric programming languages (Brainfuck, Befunge-98, Whitespace, Unlambda, and Shakespeare). All five of our chosen esoteric languages are Turing-complete, so the same algorithmic problems that are solvable in Python or JavaScript are in principle solvable in each of them. Yet, they are unfamiliar to LLMs which makes them a good proxy for evaluating out-of-distribution performance. The unfamiliarity of esoteric languages comprises of: (i) the hard-by-design primitives comprising the language; (ii) substantially less representation in pre-training corpora (340x to over 60,000x fewer public GitHub repositories than Python); (iii) negligible deployment value, which makes targeted inclusion in post-training data economically irrational. We evaluate five frontier models across five prompting strategies and find a dramatic capability gap. The same 80 problems expressed in Python or JavaScript reach 100% accuracy on top frontier models, while the equivalent esoteric versions score only 0-11%. Few-shot learning and self-reflection also fail to close this gap. EsoLang-Bench therefore provides a contamination-resistant testbed for measuring how well frontier models generalise algorithmic problem-solving to programming languages outside their training distribution.

Citations

• AlphaEvolve: A coding agent for scientific and algorithmic discovery
• Execution Guided Line-by-Line Code Generation
• Qwen3 Technical Report
• ARCS: Agentic Retrieval-Augmented Code Synthesis with Iterative Refinement
• Changing Answer Order Can Decrease MMLU Accuracy
• Benchmark Data Contamination of Large Language Models: A Survey
• A Careful Examination of Large Language Model Performance on Grade School Arithmetic
• LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
• OpenCodeInterpreter: Integrating Code Generation with Execution and Refinement
• Mathematical discoveries from program search with large language models
• Investigating Data Contamination in Modern Benchmarks for Large Language Models
• Don't Make Your LLM an Evaluation Benchmark Cheater
• NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
• Proving Test Set Contamination in Black Box Language Models
• Persistent memory for AI coding agents: a pre-registered SWE-bench Verified benchmark
• Code Llama: Open Foundation Models for Code
• Lost in the Middle: How Language Models Use Long Contexts
• Faith and Fate: Limits of Transformers on Compositionality
• Stop Uploading Test Data in Plain Text: Practical Strategies for Mitigating Data Contamination by Evaluation Benchmarks
• StarCoder: may the source be with you!
• Teaching Large Language Models to Self-Debug
• Self-Refine: Iterative Refinement with Self-Feedback
• Reflexion: Language Agents with Verbal Reinforcement Learning
• DS-1000: A Natural and Reliable Benchmark for Data Science Code Generation
• Exploring Length Generalization in Large Language Models
• InCoder: A Generative Model for Code Infilling and Synthesis
• CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis
• Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
• BNAI, NO-TOKEN, and MIND-UNITY: Pillars of a Systemic Revolution in Artificial Intelligence
• Show Your Work: Scratchpads for Intermediate Computation with Language Models
• AI and the Everything in the Whole Wide World Benchmark
• Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models
• Training Verifiers to Solve Math Word Problems
• CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation
• Program Synthesis with Large Language Models
• Evaluating Large Language Models Trained on Code
• CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP
• What Will it Take to Fix Benchmarking in Natural Language Understanding?
• Language Models are Few-Shot Learners
• Datasheets for Datasets
• Datasheets for datasets
• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks
• The Expressive Power of Transformers with Chain of Thought

Discussions

• The problem is Karsten Konrad is also wrong - LLMs really don't generalize out of their training set, and in fact there was literally just recently a paper about how they fail with novel programming l [bsky, 5 points, 4 comments]
• Por cierto, seguramente en unos días llegue por aquí este artículo, bajo titulares como "demuestran que la IA no razona, solo repite patrones memorizados" Y ya os adelanto que: es falso. La prueba se [bsky, 3 points, 2 comments]
• Evaluating Genuine Reasoning in LLMs via Esoteric Programming Languages [hn, 1 points, 1 comments]
• EsoLang-Bench arxiv.org/abs/2603.09678 In which it's made abundantly clear (yet again) that LLMs don't reason, and their effectiveness at coding is based purely on the glut of python code that's out t [bsky, 1 points, 0 comments]
• Results like this are why I think even the dumb breakdowns are valuable. This is about esolangs, but there are plenty of similarly obscure topics out there (e.g. retrocomputing): arxiv.org/abs/2603.09 [bsky, 1 points, 1 comments]
• EsoLang-Bench: Evaluating Genuine Reasoning in Large Language Models via Esoteric Programming Languages arxiv.org/abs/2603.09678 [bsky, 0 points, 0 comments]
• https://arxiv.org/abs/2603.09678 [bsky, 0 points, 0 comments]
• What is the secret to programming in Shakespeare then? arxiv.org/abs/2603.09678 [bsky, 0 points, 1 comments]
• These researchers try to get some frontier models to greenfield basic algos in languages for which they had little training data and the LLMs cannot regurgitate their way through the puzzles without t [bsky, 0 points, 1 comments]
• "Evaluating genuine reasoning in large language models via esoteric programming languages." arxiv.org/abs/2603.09678 #solidstatelife #ai #genai [bsky, 0 points, 0 comments]

Related

• Isolating Language-Coding from Problem-Solving: Benchmarking LLMs with PseudoEval
• EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code
• LEVER: Learning to Verify Language-to-Code Generation with Execution
• PythonSaga: Redefining the Benchmark to Evaluate Code Generating LLMs
• Knowledge Transfer from High-Resource to Low-Resource Programming Languages for Code LLMs