TF-Replicator: Distributed Machine Learning for Researchers

2019/02/01 by Peter Buchlovsky, Buchlovsky, Peter, David Budden +22 · 3 voices · 7 citations
Computer Science · Mathematics · #Advanced Neural Network Applications #Adversarial Robustness in Machine Learning #Artificial Intelligence (cs.AI) #Distributed #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Parallel #and Cluster Computing (cs.DC) #cs.AI #cs.DC #cs.LG #stat.ML

paper · pdf · doi:10.48550/arxiv.1902.00465

arxiv created 2019/02/01 · openalex publication_date 2019/02/01 · arxiv published 2019/02/01 · arxiv updated 2019/02/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We describe TF-Replicator, a framework for distributed machine learning designed for DeepMind researchers and implemented as an abstraction over TensorFlow. TF-Replicator simplifies writing data-parallel and model-parallel research code. The same models can be effortlessly deployed to different cluster architectures (i.e. one or many machines containing CPUs, GPUs or TPU accelerators) using synchronous or asynchronous training regimes. To demonstrate the generality and scalability of TF-Replicator, we implement and benchmark three very different models: (1) A ResNet-50 for ImageNet classification, (2) a SN-GAN for class-conditional ImageNet image generation, and (3) a D4PG reinforcement learning agent for continuous control. Our results show strong scalability performance without demanding any distributed systems expertise of the user. The TF-Replicator programming model will be open-sourced as part of TensorFlow 2.0 (see https://github.com/tensorflow/community/pull/25).

Citations

• Very Deep Convolutional Networks for Large-Scale Image Recognition
• MapReduce
• Continuous control with deep reinforcement learning
• MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems
• Improved Techniques for Training GANs
• Decoupled Neural Interfaces using Synthetic Gradients
• Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
• Mixed Precision Training
• Don't Decay the Learning Rate, Increase the Batch Size
• DeepMind Control Suite
• Spectral Normalization for Generative Adversarial Networks
• IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures
• Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes

Cited by

• Two Routes to Scalable Credit Assignment without Weight Symmetry
• Deep Learning without Weight Transport
• Non-Differentiable Supervised Learning with Evolution Strategies and Hybrid Methods
• End-to-End Adversarial Text-to-Speech
• Identifying Learning Rules From Neural Network Observables
• The Case for Strong Scaling in Deep Learning: Training Large 3D CNNs with Hybrid Parallelism
• Data Movement Is All You Need: A Case Study on Optimizing Transformers

Discussions

• TF-Replicator: Distributed Machine Learning for Researchers [hn, 3 points, 0 comments]
• TF-Replicator: Distributed Machine Learning for Researchers [hn, 2 points, 0 comments]
• TF-Replicator: Distributed Machine Learning for Researchers [hn, 1 points, 0 comments]

Related

• TF.Learn: TensorFlow's High-level Module for Distributed Machine Learning
• TensorFlow: A system for large-scale machine learning
• TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
• ChainerMN: Scalable Distributed Deep Learning Framework
• OneFlow: Redesign the Distributed Deep Learning Framework from Scratch