2021/03/12 by Muhammad Zohaib Khan, Khan, Muhammad Zohaib · 2 citations
Computer Science · Social Sciences · #Artificial Intelligence in Law #Artificial intelligence #Artificial neural network #Autoencoder #Computation and Language (cs.CL) #Computer science #Domain (mathematical analysis) #Domain adaptation #FOS: Computer and information sciences #Language model #Linguistics #Machine Learning (cs.LG) #Machine learning #Named-entity recognition #Natural Language Processing Techniques #Natural language processing #Task (project management) #Topic Modeling #Vocabulary #cs.CL #cs.LG
paper · pdf · doi:10.48550/arxiv.2103.11792
published in arXiv (Cornell University) (Cornell University)
arxiv created 2021/03/12 · openalex publication_date 2021/03/12 · arxiv updated 2021/03/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Recent developments in Natural Language Processing have led to the introduction of state-of-the-art Neural Language Models, enabled with unsupervised transferable learning, using different pretraining objectives. While these models achieve excellent results on the downstream NLP tasks, various domain adaptation techniques can improve their performance on domain-specific tasks. We compare and analyze the pretrained Neural Language Models, XLNet (autoregressive), and BERT (autoencoder) on the Legal Tasks. Results show that XLNet Model performs better on our Sequence Classification task of Legal Opinions Classification, whereas BERT produces better results on the NER task. We use domain-specific pretraining and additional legal vocabulary to adapt BERT Model further to the Legal Domain. We prepared multiple variants of the BERT Model, using both methods and their combination. Comparing our variants of the BERT Model, specializing in the Legal Domain, we conclude that both additional pretraining and vocabulary techniques enhance the BERT model's performance on the Legal Opinions Classification task. Additional legal vocabulary improves BERT's performance on the NER task. Combining the pretraining and vocabulary techniques further improves the final results. Our Legal-Vocab-BERT Model gives the best results on the Legal Opinions Task, outperforming the larger pretrained general Language Models, i.e., BERT-Base and XLNet-Base.