CoMind: Towards Community-Driven Agents for Machine Learning Engineering

2025/06/25 by Sijie Li, Li, Sijie, Weiwei Sun +7 · 2 citations
Computer Science · Decision Sciences · #Artificial Intelligence (cs.AI) #Data Stream Mining Techniques #FOS: Computer and information sciences #Machine Learning (cs.LG) #Mobile Crowdsensing and Crowdsourcing #Scientific Computing and Data Management

paper · pdf · doi:10.48550/arxiv.2506.20640

openalex publication_date 2025/06/25 · openalex created_date 2025/10/09 · openalex updated_date 2026/07/31

Abstract

Large language model (LLM) agents show promise in automating machine learning (ML) engineering. However, existing agents typically operate in isolation on a given research problem, without engaging with the broader research community, where human researchers often gain insights and contribute by sharing knowledge. To bridge this gap, we introduce MLE-Live, a live evaluation framework designed to assess an agent's ability to communicate with and leverage collective knowledge from a simulated Kaggle research community. Building on this framework, we propose CoMind, a multi-agent system designed to systematically leverage external knowledge. CoMind employs an iterative parallel exploration mechanism, developing multiple solutions simultaneously to balance exploratory breadth with implementation depth. On 75 past Kaggle competitions within our MLE-Live framework, CoMind achieves a 36% medal rate, establishing a new state of the art. Critically, when deployed in eight live, ongoing competitions, CoMind outperforms 92.6% of human competitors on average, placing in the top 5% on three official leaderboards and the top 1% on one.

Citations

• ML-Master: Towards AI-for-AI via Integration of Exploration and Reasoning
• FrontierCO: Real-World and Large-Scale Evaluation of Machine Learning Solvers for Combinatorial Optimization
• CodePDE: An Inference Framework for LLM-driven PDE Solver Generation
• DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decomposition
• The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search
• CO-Bench: Benchmarking Language Model Agents in Algorithm Search for Combinatorial Optimization
• AIDE: AI-Driven Exploration in the Space of Code
• AutoKaggle: A Multi-Agent Framework for Autonomous Data Science Competitions
• SELA: Tree-Search Enhanced LLM Agents for Automated Machine Learning
• MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
• AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML
• DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?
• Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models
• OpenHands: An Open Platform for AI Software Developers as Generalist Agents
• OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments
• Data Interpreter: An LLM Agent For Data Science
• DS-Agent: Automated Data Science by Empowering Large Language Models with Case-Based Reasoning
• Multilingual E5 Text Embeddings: A Technical Report
• Mathematical discoveries from program search with large language models
• Autonomous chemical research with large language models
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues?
• MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation
• MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework
• Autonomous Tester Agent Benchmark
• Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering
• Distributing Accountability, Not Capability: Phase Separation and the LLM Workflow Quadrant in Autonomous AI Agent Architectures
• AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data
• FLAML: A Fast and Lightweight AutoML Library
• MLPerf Training Benchmark
• DARTS: Differentiable Architecture Search
• HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face

Cited by

• Scaling Long-Horizon LLM Agent via Context-Folding
• CodePDE: An Inference Framework for LLM-driven PDE Solver Generation
• Continuous Improvement and Parallel Autonomous Exploration: An LLM-Agent Framework for Searching Large Solution Spaces

Related

• MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering
• MLE-Dojo: Interactive Environments for Empowering LLM Agents in Machine Learning Engineering
• MLE-STAR: Machine Learning Engineering Agent via Search and Targeted Refinement
• MLE-Smith: Scaling MLE Tasks with Automated Multi-Agent Pipeline
• R&D-Agent: An LLM-Agent Framework Towards Autonomous Data Science