2023/04/21 by Qiaoyue Tang, Tang, Qiaoyue, Mathias Lécuyer +1 · 3 citations
Computer Science · #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Neural Networks and Applications #Target Tracking and Data Fusion in Sensor Networks #Time Series Analysis and Forecasting
paper · pdf · doi:10.48550/arxiv.2304.11208
openalex publication_date 2023/04/21 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
We observe that the traditional use of DP with the Adam optimizer introduces a bias in the second moment estimation, due to the addition of independent noise in the gradient computation. This bias leads to a different scaling for low variance parameter updates, that is inconsistent with the behavior of non-private Adam, and Adam's sign descent interpretation. Empirically, correcting the bias introduced by DP noise significantly improves the optimization performance of DP-Adam.