2023/05/12 by Jan Hückelheim, Hückelheim, Jan, Harshitha Menon +9
Computer Science · #Artificial Intelligence (cs.AI) #Computational Engineering #Distributed and Parallel Computing Systems #FOS: Computer and information sciences #FOS: Mathematics #Finance #Numerical Analysis (math.NA) #Numerical Methods and Algorithms #Parallel Computing and Optimization Techniques #and Science (cs.CE)
paper · pdf · doi:10.48550/arxiv.2305.07546
openalex publication_date 2023/05/12 · openalex created_date 2023/05/18 · openalex updated_date 2026/08/01
Automatic differentiation, also known as backpropagation, AD, autodiff, or algorithmic differentiation, is a popular technique for computing derivatives of computer programs accurately and efficiently. Sometimes, however, the derivatives computed by AD could be interpreted as incorrect. These pitfalls occur systematically across tools and approaches. In this paper we broadly categorize problematic usages of AD and illustrate each category with examples such as chaos, time-averaged oscillations, discretizations, fixed-point loops, lookup tables, and linear solvers. We also review debugging techniques and their effectiveness in these situations. With this article we hope to help readers avoid unexpected behavior, detect problems more easily when they occur, and have more realistic expectations from AD tools.