2020/06/17 by Ravi G. Patel, Patel, Ravi G., Nathaniel Trask +5 · 1 citation
Computer Science · Engineering · #3D Shape Modeling and Analysis #Advanced Numerical Analysis Techniques #FOS: Computer and information sciences #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Optical measurement and interference techniques
paper · pdf · doi:10.48550/arxiv.2006.10123
openalex publication_date 2020/06/17 · openalex created_date 2022/07/26 · openalex updated_date 2026/07/28
Second-order optimizers hold intriguing potential for deep learning, but\nsuffer from increased cost and sensitivity to the non-convexity of the loss\nsurface as compared to gradient-based approaches. We introduce a coordinate\ndescent method to train deep neural networks for classification tasks that\nexploits global convexity of the cross-entropy loss in the weights of the\nlinear layer. Our hybrid Newton/Gradient Descent (NGD) method is consistent\nwith the interpretation of hidden layers as providing an adaptive basis and the\nlinear layer as providing an optimal fit of the basis to data. By alternating\nbetween a second-order method to find globally optimal parameters for the\nlinear layer and gradient descent to train the hidden layers, we ensure an\noptimal fit of the adaptive basis to data throughout training. The size of the\nHessian in the second-order step scales only with the number weights in the\nlinear layer and not the depth and width of the hidden layers; furthermore, the\napproach is applicable to arbitrary hidden layer architecture. Previous work\napplying this adaptive basis perspective to regression problems demonstrated\nsignificant improvements in accuracy at reduced training cost, and this work\ncan be viewed as an extension of this approach to classification problems. We\nfirst prove that the resulting Hessian matrix is symmetric semi-definite, and\nthat the Newton step realizes a global minimizer. By studying classification of\nmanufactured two-dimensional point cloud data, we demonstrate both an\nimprovement in validation error and a striking qualitative difference in the\nbasis functions encoded in the hidden layer when trained using NGD. Application\nto image classification benchmarks for both dense and convolutional\narchitectures reveals improved training accuracy, suggesting possible gains of\nsecond-order methods over gradient descent.\n