2021/08/03 by Rohan Anil, Badih Ghazi, Anil, Rohan +7 · 7 citations
Computer Science · Engineering · #Computation and Language (cs.CL) #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #Ferroelectric and Negative Capacitance Devices #Machine Learning (cs.LG) #Natural Language Processing Techniques #Topic Modeling #cs.CL #cs.CR #cs.LG
paper · pdf · doi:10.48550/arxiv.2108.01624
12 pages, 6 figures
arxiv created 2021/08/03 · openalex publication_date 2021/08/03 · arxiv updated 2021/08/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
In this work, we study the large-scale pretraining of BERT-Large with differentially private SGD (DP-SGD). We show that combined with a careful implementation, scaling up the batch size to millions (i.e., mega-batches) improves the utility of the DP-SGD step for BERT; we also enhance its efficiency by using an increasing batch size schedule. Our implementation builds on the recent work of [SVK20], who demonstrated that the overhead of a DP-SGD step is minimized with effective use of JAX [BFH+18, FJL18] primitives in conjunction with the XLA compiler [XLA17]. Our implementation achieves a masked language model accuracy of 60.5% at a batch size of 2M, for ε= 5.36. To put this number in perspective, non-private BERT models achieve an accuracy of ∼70%.