Zerrow: True Zero-Copy Arrow Pipelines in Bauplan

2025/04/08 by Yifan Dai, Jacopo Tagliabue, Dai, Yifan +9 · 1 voice · 1 citation
Computer Science · Decision Sciences · #Advanced Database Systems and Queries #Databases (cs.DB) #FOS: Computer and information sciences #Operating Systems (cs.OS) #Scientific Computing and Data Management #Security and Verification in Computing #cs.DB #cs.OS

paper · pdf · doi:10.48550/arxiv.2504.06151

openalex publication_date 2025/04/08 · arxiv published 2025/04/08 · arxiv updated 2025/05/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Bauplan is a FaaS-based lakehouse specifically built for data pipelines: its execution engine uses Apache Arrow for data passing between the nodes in the DAG. While Arrow is known as the "zero copy format", in practice, limited Linux kernel support for shared memory makes it difficult to avoid copying entirely. In this work, we introduce several new techniques to eliminate nearly all copying from pipelines: in particular, we implement a new kernel module that performs de-anonymization, thus eliminating a copy to intermediate data. We conclude by sharing our preliminary evaluation on different workloads types, as well as discussing our plan for future improvements.

Citations

• Bauplan: zero-copy, scale-up FaaS for data pipelines
• Reproducible data science over data lakes: replayable data pipelines with Bauplan and Nessie
• Building a serverless Data Lakehouse from spare parts
• Reasonable Scale Machine Learning with Open-Source Metaflow
• You Do Not Need a Bigger Boat: Recommendations at Reasonable Scale in a (Mostly) Serverless and Open Stack
• SIGIR 2021 E-Commerce Workshop Data Challenge
• Starling: A Scalable Query Engine on Cloud Function Services
• How Developers Iterate on Machine Learning Workflows -- A Survey of the Applied Machine Learning Literature
• MapReduce

Discussions

• Yifan Dai, Jacopo Tagliabue, Andrea Arpaci-Dusseau, Remzi Arpaci-Dusseau, Tyler R. Caraza-Harter: Zerrow: True Zero-Copy Arrow Pipelines in Bauplan https://arxiv.org/abs/2504.06151 https://arxiv.org/p [bsky, 0 points, 1 comments]

Related

• Leveraging Apache Arrow for Zero-copy, Zero-serialization Cluster Shared Memory
• Bauplan: zero-copy, scale-up FaaS for data pipelines
• Defeat the Heap: Zero-Copy Data Movement in AXI4MLIR
• Building a serverless Data Lakehouse from spare parts
• Skyhook: Towards an Arrow-Native Storage System