Do Large Language Models Speak Scientific Workflows?

2024/12/13 by Orçun Yildiz, Tom Peterka, Yildiz, Orcun +1
Computer Science · Decision Sciences · #FOS: Computer and information sciences #Human-Computer Interaction (cs.HC) #Research Data Management Practices #Scientific Computing and Data Management

paper · pdf · doi:10.48550/arxiv.2412.10606

openalex publication_date 2024/12/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/31

Abstract

With the advent of large language models (LLMs), there is a growing interest in applying LLMs to scientific tasks. In this work, we conduct an experimental study to explore applicability of LLMs for configuring, annotating, translating, explaining, and generating scientific workflows. We use 5 different workflow specific experiments and evaluate several open- and closed-source language models using state-of-the-art workflow systems. Our studies reveal that LLMs often struggle with workflow related tasks due to their lack of knowledge of scientific workflows. We further observe that the performance of LLMs varies across experiments and workflow systems. Our findings can help workflow developers and users in understanding LLMs capabilities in scientific workflows, and motivate further research applying LLMs to workflows.

Related

• How should the advent of large language models affect the practice of science?
• Advancing the Scientific Method with Large Language Models: From Hypothesis to Discovery
• On the effectiveness of Large Language Models for GitHub Workflows
• The unintended consequences of large language models as a labor-augmenting technology in science
• Evolving Roles of LLMs in Scientific Innovation: Assistant, Collaborator, Scientist, and Evaluator