2018/03/29 by Barbara Barabasz, A. W. Anderson, Barabasz, Barbara +5 · 1 citation
Computer Science · Physics and Astronomy · #Numerical Methods and Algorithms #Model Reduction and Neural Networks #Neural Networks and Applications
paper · pdf · doi:10.48550/arxiv.1803.10986
Popular deep neural networks (DNNs) spend the majority of their execution\ntime computing convolutions. The Winograd family of algorithms can greatly\nreduce the number of arithmetic operations required and is present in many DNN\nsoftware frameworks. However, the performance gain is at the expense of a\nreduction in floating point (FP) numerical accuracy. In this paper, we analyse\nthe worst case FP error and prove the estimation of norm and conditioning of\nthe algorithm. We show that the bound grows exponentially with the size of the\nconvolution, but the error bound of the \modified algorithm is smaller\nthan the original one. We propose several methods for reducing FP error. We\npropose a canonical evaluation ordering based on Huffman coding that reduces\nsummation error. We study the selection of sampling "points" experimentally and\nfind empirically good points for the most important sizes. We identify the main\nfactors associated with good points. In addition, we explore other methods to\nreduce FP error, including mixed-precision convolution, and pairwise summation\nacross DNN channels. Using our methods we can significantly reduce FP error for\na given block size, which allows larger block sizes and reduced computation.\n