2021/10/13 by Zhuosheng Zhang, Zhang, Zhuosheng, Hanqing Zhang +11 · 30 citations
Computer Science · Engineering · Mathematics · #Algorithm #Artificial Intelligence (cs.AI) #Artificial intelligence #Benchmark (surveying) #Computation #Computation and Language (cs.CL) #Computer science #Discriminative model #Domain (mathematical analysis) #Engineering #FOS: Computer and information sciences #Generative grammar #Key (lock) #Machine learning #Mathematics #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Range (aeronautics) #Software deployment #Software engineering #Topic Modeling #cs.AI #cs.CL
paper · pdf · doi:10.48550/arxiv.2110.06696
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2021/10/13 · arxiv created 2021/10/14 · arxiv updated 2021/10/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Although pre-trained models (PLMs) have achieved remarkable improvements in a wide range of NLP tasks, they are expensive in terms of time and resources. This calls for the study of training more efficient models with less computation but still ensures impressive performance. Instead of pursuing a larger scale, we are committed to developing lightweight yet more powerful models trained with equal or less computation and friendly to rapid deployment. This technical report releases our pre-trained model called Mengzi, which stands for a family of discriminative, generative, domain-specific, and multimodal pre-trained model variants, capable of a wide range of language and vision tasks. Compared with public Chinese PLMs, Mengzi is simple but more powerful. Our lightweight model has achieved new state-of-the-art results on the widely-used CLUE benchmark with our optimized pre-training and fine-tuning techniques. Without modifying the model architecture, our model can be easily employed as an alternative to existing PLMs. Our sources are available at https://github.com/Langboat/Mengzi.