ML-Dev-Bench: Comparative Analysis of AI Agents on ML development workflows

2025/02/03 by Harshith Padigela, Chintan Shah, Padigela, Harshith +3 · 7 citations
Computer Science · #Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Multi-Agent Systems and Negotiation #Software Engineering (cs.SE)

paper · pdf · doi:10.48550/arxiv.2502.00964

openalex publication_date 2025/02/03 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

In this report, we present ML-Dev-Bench, a benchmark aimed at testing agentic capabilities on applied Machine Learning development tasks. While existing benchmarks focus on isolated coding tasks or Kaggle-style competitions, ML-Dev-Bench tests agents' ability to handle the full complexity of ML development workflows. The benchmark assesses performance across critical aspects including dataset handling, model training, improving existing models, debugging, and API integration with popular ML tools. We evaluate three agents - ReAct, Openhands, and AIDE - on a diverse set of 30 tasks, providing insights into their strengths and limitations in handling practical ML development challenges. We open source the benchmark for the benefit of the community at \hrefhttps://github.com/ml-dev-bench/ml-dev-benchhttps://github.com/ml-dev-bench/ml-dev-bench.

Cited by

• ReX-MLE: The Autonomous Agent Benchmark for Medical Imaging Challenges
• FML-bench: A Benchmark for Automatic ML Research Agents Highlighting the Importance of Exploration Breadth
• RECODE-H: A Benchmark for Research Code Development with Interactive Human Feedback
• ML2B: Benchmarking LLMs on Cross-Lingual ML Pipeline Generation
• ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agents
• Large Language Model-based Data Science Agent: A Survey
• How Far Are AI Scientists from Changing the World?

Related

• MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
• ML-Tool-Bench: Tool-Augmented Planning for ML Tasks
• ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code
• ML Research Benchmark
• Towards Adaptive ML Benchmarks: Web-Agent-Driven Construction, Domain Expansion, and Metric Optimization