Node-Aware Improvements to Allreduce

2019/10/21 by Bienz, Amanda, Olson, Luke N., Gropp, William D. · 1 citation
#Distributed #FOS: Computer and information sciences #Parallel #and Cluster Computing (cs.DC)

paper · doi:10.48550/arxiv.1910.09650

Abstract

The MPI_Allreduce collective operation is a core kernel of many parallel codebases, particularly for reductions over a single value per process. The commonly used allreduce recursive-doubling algorithm obtains the lower bound message count, yielding optimality for small reduction sizes based on node-agnostic performance models. However, this algorithm yields duplicate messages between sets of nodes. Node-aware optimizations in MPICH remove duplicate messages through use of a single master process per node, yielding a large number of inactive processes at each inter-node step. In this paper, we present an algorithm that uses the multiple processes available per node to reduce the maximum number of inter-node messages communicated by a single process, improving the performance of allreduce operations, particularly for small message sizes.

Cited by

• Optimizing Allreduce Operations for Modern Heterogeneous Architectures with Multiple Processes per GPU

Related

• A Doubly-pipelined, Dual-root Reduction-to-all Algorithm and Implementation
• Optimal, Non-pipelined Reduce-scatter and Allreduce Algorithms
• A Locality-Aware Bruck Allgather
• Optimised allgatherv, reduce_scatter and allreduce communication in message-passing systems
• A Generalization of the Allreduce Operation