2021/07/27 by Supun Kamburugamuve, Kamburugamuve, Supun, Chathura Widanage +13
Computer Science · #Artificial Intelligence (cs.AI) #Computational Physics and Python Applications #Distributed #FOS: Computer and information sciences #Parallel #Parallel Computing and Optimization Techniques #Software System Performance and Reliability #and Cluster Computing (cs.DC)
paper · pdf · doi:10.48550/arxiv.2107.12807
openalex publication_date 2021/07/27 · openalex created_date 2022/07/25 · openalex updated_date 2026/07/28
Data-intensive applications impact many domains, and their steadily\nincreasing size and complexity demands high-performance, highly usable\nenvironments. We integrate a set of ideas developed in various data science and\ndata engineering frameworks. They employ a set of operators on specific data\nabstractions that include vectors, matrices, tensors, graphs, and tables. Our\nkey concepts are inspired from systems like MPI, HPF (High-Performance\nFortran), NumPy, Pandas, Spark, Modin, PyTorch, TensorFlow, RAPIDS(NVIDIA), and\nOneAPI (Intel). Further, it is crucial to support different languages in\neveryday use in the Big Data arena, including Python, R, C++, and Java. We note\nthe importance of Apache Arrow and Parquet for enabling language agnostic high\nperformance and interoperability. In this paper, we propose High-Performance\nTensors, Matrices and Tables (HPTMT), an operator-based architecture for\ndata-intensive applications, and identify the fundamental principles needed for\nperformance and usability success. We illustrate these principles by a\ndiscussion of examples using our software environments, Cylon and Twister2 that\nembody HPTMT.\n