2020/06/18 by Mauricio Tano, Tano, Mauricio E., Gavin Portwood +3
Engineering · #Refrigeration and Air Conditioning Technologies #Machine Fault Diagnosis Techniques #Cavitation Phenomena in Pumps
paper · pdf · doi:10.48550/arxiv.2006.14371
Training of deep neural networks (DNNs) frequently involves optimizing\nseveral millions or even billions of parameters. Even with modern computing\narchitectures, the computational expense of DNN training can inhibit, for\ninstance, network architecture design optimization, hyper-parameter studies,\nand integration into scientific research cycles. The key factor limiting\nperformance is that both the feed-forward evaluation and the back-propagation\nrule are needed for each weight during optimization in the update rule. In this\nwork, we propose a method to decouple the evaluation of the update rule at each\nweight. At first, Proper Orthogonal Decomposition (POD) is used to identify a\ncurrent estimate of the principal directions of evolution of weights per layer\nduring training based on the evolution observed with a few backpropagation\nsteps. Then, Dynamic Mode Decomposition (DMD) is used to learn the dynamics of\nthe evolution of the weights in each layer according to these principal\ndirections. The DMD model is used to evaluate an approximate converged state\nwhen training the ANN. Afterward, some number of backpropagation steps are\nperformed, starting from the DMD estimates, leading to an update to the\nprincipal directions and DMD model. This iterative process is repeated until\nconvergence. By fine-tuning the number of backpropagation steps used for each\nDMD model estimation, a significant reduction in the number of operations\nrequired to train the neural networks can be achieved. In this paper, the DMD\nacceleration method will be explained in detail, along with the theoretical\njustification for the acceleration provided by DMD. This method is illustrated\nusing a regression problem of key interest for the scientific machine learning\ncommunity: the prediction of a pollutant concentration field in a diffusion,\nadvection, reaction problem.\n