What Twelve LLM Agent Benchmark Papers Disclose About Themselves: A Pilot Audit and an Open Scoring Schema

2026/05/20 by Mahdi Naser Moghadasi, Faezeh Ghaderi · 1 voice · 5 citations
Business, Management and Accounting · Computer Science · Social Sciences · #Accounting Education and Careers #Artificial Intelligence in Law #Audit #Auditing, Earnings Management, Governance #Benchmark (surveying) #Benchmarking #Inference #Schema (genetic algorithms) #cs.LG

paper · pdf · open access · doi:10.48550/arxiv.2605.21404

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2026/05/20 · arxiv published 2026/05/20 · arxiv updated 2026/05/20 · openalex created_date 2026/05/22 · openalex updated_date 2026/07/29

Abstract

We read twelve well-known LLM agent benchmark papers and recorded, dimension by dimension, what each paper actually says about how its evaluation was run. The motivation came from a familiar frustration: two papers will report results on the same benchmark with the same model name and disagree, and you cannot tell why -- the scaffold, the sampling settings, the subset, or the evaluator version. In many cases the published artifact does not let you answer. This paper is an implementation report on the attempt. We designed a small audit schema (five fields: benchmark identity, harness specification, inference settings, cost reporting, failure breakdown), wrote a scoring codebook with the boundary cases we hit during pilot scoring, applied it to twelve canonical papers (eight agent, four classical static), and recorded what we saw. We score the disclosure of an agent run, not its correctness, and make no claim that disclosure implies a trustworthy result. The mean audit score across the eight agent-benchmark papers is 0.38 (out of 1.0), and across the four classical static benchmarks 0.66; the largest gap is on cost (none of the eight agent benchmark papers disclose inference cost in any form) and on harness specification (none fully disclose a content-addressed container image of the evaluation environment). We release the schema as a JSON Schema file, the codebook as a Markdown document, and the raw scoring sheet as a CSV. The scoring was performed by a single auditor in one pass; a multi-rater audit is the natural next step, and we discuss what we think it would change.

Citations

• BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices
• MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
• OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
• AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents
• State of What Art? A Call for Multi-Prompt LLM Evaluation
• GAIA: a benchmark for General AI Assistants
• Proving Test Set Contamination in Black Box Language Models
• Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• AgentBench: Evaluating LLMs as Agents
• Autonomous Tester Agent Benchmark
• Mind2Web: Towards a Generalist Agent for the Web
• Data Contamination: From Memorization to Exploitation
• Training Verifiers to Solve Math Word Problems
• Program Synthesis with Large Language Models
• Evaluating Large Language Models Trained on Code
• Measuring Massive Multitask Language Understanding
• Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)
• Model Cards for Model Reporting
• Datasheets for Datasets
• Deep Reinforcement Learning that Matters
• VisualWebArena: Evaluating Multimodal Agents on Realistic Visual Web Tasks

Discussions

• 12 papers, same benchmark — different, incomparable experimental conditions. Scaffolding, judge prompts, environment versioning: rarely disclosed. You can't trust leaderboard rankings when scores aren [bsky, 0 points, 1 comments]

Related

• Standard Benchmarks Fail -- Auditing LLM Agents in Finance Must Prioritize Risk
• Evaluating and Guarding Citation Faithfulness in Agentic Scientific Synthesis
• When Judgment Becomes Noise: How Design Failures in LLM Judge Benchmarks Silently Undermine Validity
• Don't Make Your LLM an Evaluation Benchmark Cheater
• HALLMARK: Diagnosing Three Failure Modes in LLM Citation Verifiers