2021/12/12 by Tanya Roosta, Roosta, Tanya, Peyman Passban +3
Computer Science · Engineering · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Ferroelectric and Negative Capacitance Devices #Privacy-Preserving Technologies in Data
paper · pdf · doi:10.48550/arxiv.2112.06135
openalex publication_date 2021/12/12 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Training neural machine translation (NMT) models in federated learning (FL) settings could be inefficient both computationally and communication-wise, due to the large size of translation engines as well as the multiple rounds of updates required to train clients and a central server. In this paper, we explore how to efficiently build NMT models in an FL setup by proposing a novel solution. In order to reduce the communication overhead, out of all neural layers we only exchange what we term "Controller" layers. Controllers are a small number of additional neural components connected to our pre-trained architectures. These new components are placed in between original layers. They act as liaisons to communicate with the central server and learn minimal information that is sufficient enough to update clients. We evaluated the performance of our models on five datasets from different domains to translate from German into English. We noted that the models equipped with Controllers preform on par with those trained in a central and non-FL setting. In addition, we observed a substantial reduction in the communication traffic of the FL pipeline, which is a direct consequence of using Controllers. Based on our experiments, Controller-based models are ~6 times less expensive than their other peers. This reduction is significantly important when we consider the number of parameters in large models and it becomes even more critical when such parameters need to be exchanged for multiple rounds in FL settings.