2021/10/15 by Iacopo Colonnelli, Marco Aldinucci, Barbara Cantalupo +8 · 1 citation
Computer Science · Decision Sciences · #Cloud Computing and Resource Management #Distributed and Parallel Computing Systems #Scientific Computing and Data Management
paper · doi:10.1016/j.future.2021.10.007
openalex publication_date 2021/10/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/30
The designers of a new coordination interface enacting complex workflows have to tackle a dichotomy: choosing a language-independent or language-dependent approach. Language-independent approaches decouple workflow models from the host code’s business logic and advocate portability. Language-dependent approaches foster flexibility and performance by adopting the same host language for business and coordination code. Jupyter Notebooks, with their capability to describe both imperative and declarative code in a unique format, allow taking the best of the two approaches, maintaining a clear separation between application and coordination layers but still providing a unified interface to both aspects. We advocate the Jupyter Notebooks’ potential to express complex distributed workflows, identifying the general requirements for a Jupyter-based Workflow Management System (WMS) and introducing a proof-of-concept portable implementation working on hybrid Cloud-HPC infrastructures. As a byproduct, we extended the vanilla IPython kernel with workflow-based parallel and distributed execution capabilities. The proposed Jupyter-workflow (Jw) system is evaluated on common scenarios for High Performance Computing (HPC) and Cloud, showing its potential in lowering the barriers between prototypical Notebooks and production-ready implementations.