2024/04/17 by Haotian Chen, Chen, Haotian, Xinjie Shen +13
Computer Science · Decision Sciences · Engineering · #Artificial Intelligence (cs.AI) #Distributed and Parallel Computing Systems #FOS: Computer and information sciences #FOS: Economics and business #General Finance (q-fin.GN) #Reservoir Engineering and Simulation Methods #Scientific Computing and Data Management
paper · pdf · doi:10.48550/arxiv.2404.11276
openalex publication_date 2024/04/17 · openalex created_date 2024/04/19 · openalex updated_date 2026/07/28
The progress of humanity is driven by those successful discoveries accompanied by countless failed experiments. Researchers often seek the potential research directions by reading and then verifying them through experiments. The process imposes a significant burden on researchers. In the past decade, the data-driven black-box deep learning method has demonstrated its effectiveness in a wide range of real-world scenarios, which exacerbates the experimental burden of researchers and thus renders the potential successful discoveries veiled. Therefore, automating such a research and development (R&D) process is an urgent need. In this paper, we serve as the first effort to formalize the goal by proposing a Real-world Data-centric automatic R&D Benchmark, namely RD2Bench. RD2Bench benchmarks all the operations in data-centric automatic R&D (D-CARD) as a whole to navigate future work toward our goal directly. We focus on evaluating the interaction and synergistic effects of various model capabilities and aiding in selecting well-performing trustworthy models. Although RD2Bench is very challenging to the state-of-the-art (SOTA) large language model (LLM) named GPT-4, indicating ample research opportunities and more research efforts, LLMs possess promising potential to bring more significant development to D-CARD: They are able to implement some simple methods without adopting any additional techniques. We appeal to future work to take developing techniques for tackling automatic R&D into consideration, thus bringing the opportunities of the potential revolutionary upgrade to human productivity.