2015/08/01 by Andrew Crotty, Alex Galakatos, Kayhan Dursun +4 · 2 citations
Decision Sciences · Computer Science · #Scientific Computing and Data Management #Advanced Database Systems and Queries #Cloud Computing and Resource Management
paper · doi:10.14778/2824032.2824045
openalex publication_date 2015/08/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/31
Data analytics has recently grown to include increasingly sophisticated techniques, such as machine learning and advanced statistics. Users frequently express these complex analytics tasks as workflows of user-defined functions (UDFs) that specify each algorithmic step. However, given typical hardware configurations and dataset sizes, the core challenge of complex analytics is no longer sheer data volume but rather the computation itself, and the next generation of analytics frameworks must focus on optimizing for this computation bottleneck. While query compilation has gained widespread popularity as a way to tackle the computation bottleneck for traditional SQL workloads, relatively little work addresses UDF-centric workflows in the domain of complex analytics. In this paper, we describe a novel architecture for automatically compiling workflows of UDFs. We also propose several optimizations that consider properties of the data, UDFs, and hardware together in order to generate different code on a case-by-case basis. To evaluate our approach, we implemented these techniques in T upleware , a new high-performance distributed analytics system, and our benchmarks show performance improvements of up to three orders of magnitude compared to alternative systems.