2022/12/20 by Amir Shaikhha, Mathieu Huot, Shaikhha, Amir +9
Computer Science · Engineering · Mathematics · #Advanced Control Systems Optimization #Advanced Optimization Algorithms Research #FOS: Computer and information sciences #Machine Learning (cs.LG) #Mathematical Software (cs.MS) #Numerical Methods and Algorithms #Programming Languages (cs.PL)
paper · pdf · doi:10.48550/arxiv.2212.10307
openalex publication_date 2022/12/20 · openalex created_date 2023/01/04 · openalex updated_date 2026/07/28
Automatic differentiation (AD) is a technique for computing the derivative of a function represented by a program. This technique is considered as the de-facto standard for computing the differentiation in many machine learning and optimisation software tools. Despite the practicality of this technique, the performance of the differentiated programs, especially for functional languages and in the presence of vectors, is suboptimal. We present an AD system for a higher-order functional array-processing language. The core functional language underlying this system simultaneously supports both source-to-source forward-mode AD and global optimisations such as loop transformations. In combination, gradient computation with forward-mode AD can be as efficient as reverse mode, and the Jacobian matrices required for numerical algorithms such as Gauss-Newton and Levenberg-Marquardt can be efficiently computed.