2024/06/18 by Zhaoxian Wu, Tayfun Gokmen, Wu, Zhaoxian +5 · 2 citations
Computer Science · Engineering · #Advanced Memory and Neural Computing #FOS: Computer and information sciences #FOS: Mathematics #Ferroelectric and Negative Capacitance Devices #Hardware Architecture (cs.AR) #Machine Learning (cs.LG) #Neural Networks and Applications #Optimization and Control (math.OC)
paper · pdf · doi:10.48550/arxiv.2406.12774
openalex publication_date 2024/06/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Given the high economic and environmental costs of using large vision or language models, analog in-memory accelerators present a promising solution for energy-efficient AI. While inference on analog accelerators has been studied recently, the training perspective is underexplored. Recent studies have shown that the "workhorse" of digital AI training - stochastic gradient descent (SGD) algorithm converges inexactly when applied to model training on non-ideal devices. This paper puts forth a theoretical foundation for gradient-based training on analog devices. We begin by characterizing the non-convergent issue of SGD, which is caused by the asymmetric updates on the analog devices. We then provide a lower bound of the asymptotic error to show that there is a fundamental performance limit of SGD-based analog training rather than an artifact of our analysis. To address this issue, we study a heuristic analog algorithm called Tiki-Taka that has recently exhibited superior empirical performance compared to SGD and rigorously show its ability to exactly converge to a critical point and hence eliminates the asymptotic error. The simulations verify the correctness of the analyses.