dMath: Distributed Linear Algebra for DL

2016/11/19 by Steven Eliuk, Cameron Upright, Eliuk, Steven +7 · 1 citation
Computer Science · #Algorithms and Data Compression #Distributed #FOS: Computer and information sciences #Mathematical Software (cs.MS) #Neural Networks and Applications #Neural and Evolutionary Computing (cs.NE) #Parallel #Parallel Computing and Optimization Techniques #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.1611.07819

openalex publication_date 2016/11/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

The paper presents a parallel math library, dMath, that demonstrates leading scaling when using intranode, internode, and hybrid-parallelism for deep learning (DL). dMath provides easy-to-use distributed primitives and a variety of domain-specific algorithms including matrix multiplication, convolutions, and others allowing for rapid development of scalable applications like deep neural networks (DNNs). Persistent data stored in GPU memory and advanced memory management techniques avoid costly transfers between host and device. dMath delivers performance, portability, and productivity to its specific domain of support.

Citations

• Kaldi Speech Recognition Toolkit

Cited by

• Jointly Optimizing Preprocessing and Inference for DNN-based Visual Analytics

Related

• dMath: A Scalable Linear Algebra and Math Library for Heterogeneous\n GP-GPU Architectures
• cuDNN: Efficient Primitives for Deep Learning
• A Linear Algebraic Approach to Model Parallelism in Deep Learning
• AutoDDL: Automatic Distributed Deep Learning with Near-Optimal Bandwidth Cost
• PolyDL: Polyhedral Optimizations for Creation of High Performance DL primitives