MLRC-Bench: Can Language Agents Solve Machine Learning Research Challenges?

2025/04/13 by Yunxiang Zhang, Muhammad Khalifa, Zhang, Yunxiang +14 · 4 citations
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2504.09702

openalex publication_date 2025/04/13 · openalex created_date 2025/10/14 · openalex updated_date 2026/07/28

Abstract

We introduce MLRC-Bench, a benchmark designed to quantify how effectively language agents can tackle challenging Machine Learning (ML) Research Competitions, with a focus on open research problems that demand novel methodologies. Unlike prior work, e.g., AI Scientist, which evaluates the end-to-end agentic pipeline by using LLM-as-a-judge, MLRC-Bench measures the key steps of proposing and implementing novel research methods and evaluates them with rigorous protocol and objective metrics. Our curated suite of 7 competition tasks reveals significant challenges for LLM agents. Even the best-performing tested agent (gemini-exp-1206 under MLAB) closes only 9.3% of the gap between baseline and top human participant scores. Furthermore, our analysis reveals a misalignment between the LLM-judged innovation and actual performance on cutting-edge ML research problems. MLRC-Bench is a dynamic benchmark, designed to grow with new ML competitions and encourage rigorous, objective evaluations of AI research capabilities. Our leaderboard and code are available at: https://huggingface.co/spaces/launch/MLRC_Bench

Cited by

• Mimosa Framework: Toward Evolving Multi-Agent Systems for Scientific Research
• One Run Is Not an Idea: The Implementation Lottery in Automated Research
• Automated Transcript Analysis for Detecting Flaws in Agentic Benchmarks
• Reinforcement Learning for Machine Learning Engineering Agents

Related

• MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research
• FML-bench: A Benchmark for Automatic ML Research Agents Highlighting the Importance of Exploration Breadth
• ML Research Benchmark
• MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
• AI Research Agents for Machine Learning: Search, Exploration, and Generalization in MLE-bench