2021/11/03 by Ehsan Tavan, Tavan, Ehsan, Ali Rahmati +7
Computer Science · #Algorithm #Artificial intelligence #Computation and Language (cs.CL) #Computer science #Deep learning #Encoder #FOS: Computer and information sciences #Language model #Natural Language Processing Techniques #Natural language processing #Pooling #Residual #Sentence #Speech recognition #Text Readability and Simplification #Topic Modeling #cs.CL
paper · pdf · doi:10.48550/arxiv.2111.02188
openalex publication_date 2021/11/03 · arxiv created 2021/11/04 · arxiv updated 2021/11/05 · openalex created_date 2022/05/05 · openalex updated_date 2026/08/05
This paper presents a deep neural architecture, for Natural Language Sentence Matching (NLSM) by adding a deep recursive encoder to BERT so called BERT with Deep Recursive Encoder (BERT-DRE). Our analysis of model behavior shows that BERT still does not capture the full complexity of text, so a deep recursive encoder is applied on top of BERT. Three Bi-LSTM layers with residual connection are used to design a recursive encoder and an attention module is used on top of this encoder. To obtain the final vector, a pooling layer consisting of average and maximum pooling is used. We experiment our model on four benchmarks, SNLI, FarsTail, MultiNLI, SciTail, and a novel Persian religious questions dataset. This paper focuses on improving the BERT results in the NLSM task. In this regard, comparisons between BERT-DRE and BERT are conducted, and it is shown that in all cases, BERT-DRE outperforms BERT. The BERT algorithm on the religious dataset achieved an accuracy of 89.70%, and BERT-DRE architectures improved to 90.29% using the same dataset.