2018/06/04 by Cody Coleman, Daniel Kang, Coleman, Cody +17 · 1 citation
Computer Science · Materials Science · #Advanced Neural Network Applications #Parallel Computing and Optimization Techniques #Machine Learning in Materials Science
paper · pdf · doi:10.48550/arxiv.1806.01427
Researchers have proposed hardware, software, and algorithmic optimizations\nto improve the computational performance of deep learning. While some of these\noptimizations perform the same operations faster (e.g., increasing GPU clock\nspeed), many others modify the semantics of the training procedure (e.g.,\nreduced precision), and can impact the final model's accuracy on unseen data.\nDue to a lack of standard evaluation criteria that considers these trade-offs,\nit is difficult to directly compare these optimizations. To address this\nproblem, we recently introduced DAWNBench, a benchmark competition focused on\nend-to-end training time to achieve near-state-of-the-art accuracy on an unseen\ndataset---a combined metric called time-to-accuracy (TTA). In this work, we\nanalyze the entries from DAWNBench, which received optimized submissions from\nmultiple industrial groups, to investigate the behavior of TTA as a metric as\nwell as trends in the best-performing entries. We show that TTA has a low\ncoefficient of variation and that models optimized for TTA generalize nearly as\nwell as those trained using standard methods. Additionally, even though\nDAWNBench entries were able to train ImageNet models in under 3 minutes, we\nfind they still underutilize hardware capabilities such as Tensor Cores.\nFurthermore, we find that distributed entries can spend more than half of their\ntime on communication. We show similar findings with entries to the MLPERF v0.5\nbenchmark.\n