2022/10/24 by Luca Di Liello, Di Liello, Luca, Matteo Gabburo +3
Computer Science · #Anomaly Detection Techniques and Applications #Computation and Language (cs.CL) #FOS: Computer and information sciences #Music and Audio Processing #Neural Networks and Applications
paper · pdf · doi:10.48550/arxiv.2210.13536
openalex publication_date 2022/10/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
In this paper, we study trade-offs between efficiency, cost and accuracy when pre-training Transformer encoders with different pre-training objectives. For this purpose, we analyze features of common objectives and combine them to create new effective pre-training approaches. Specifically, we designed light token generators based on a straightforward statistical approach, which can replace ELECTRA computationally heavy generators, thus highly reducing cost. Our experiments also show that (i) there are more efficient alternatives to BERT's MLM, and (ii) it is possible to efficiently pre-train Transformer-based models using lighter generators without a significant drop in performance.