2020/11/17 by Wei Zhu, Zhu, Wei · 4 citations
Computer Science · #Artificial intelligence #Computation and Language (cs.CL) #Computer science #FOS: Computer and information sciences #Lexical analysis #Linguistics #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Natural language #Natural language processing #Natural language understanding #Sentence #Sequence labeling #Speech recognition #Task (project management) #Topic Modeling #Vocabulary #Word (group theory) #cs.CL
paper · pdf · doi:10.48550/arxiv.2011.08539
published in arXiv (Cornell University) (Cornell University)
arxiv created 2020/11/17 · openalex publication_date 2020/11/17 · arxiv updated 2020/11/18 · openalex created_date 2020/11/23 · openalex updated_date 2026/08/06
Despite the development of pre-trained language models (PLMs) significantly raise the performances of various Chinese natural language processing (NLP) tasks, the vocabulary for these Chinese PLMs remain to be the one provided by Google Chinese Bert \citedevlin2018bert, which is based on Chinese characters. Second, the masked language model pre-training is based on a single vocabulary, which limits its downstream task performances. In this work, we first propose a novel method, seg_tok, to form the vocabulary of Chinese BERT, with the help of Chinese word segmentation (CWS) and subword tokenization. Then we propose three versions of multi-vocabulary pretraining (MVP) to improve the models expressiveness. Experiments show that: (a) compared with char based vocabulary, seg_tok does not only improves the performances of Chinese PLMs on sentence level tasks, it can also improve efficiency; (b) MVP improves PLMs' downstream performance, especially it can improve seg_tok's performances on sequence labeling tasks.