2021/01/01 by Xiang Hu, Haitao Mi, Zujie Wen +4
Computer Science · Engineering · Mathematics · Social Sciences · #Artificial intelligence #Combinatorics #Computational and Text Analysis Methods #Computer science #Differentiable function #Electrical engineering #Engineering #Mathematics #Natural Language Processing Techniques #Natural language processing #Programming language #Pure mathematics #Theoretical computer science #Topic Modeling #Transformer #Tree (set theory) #cs.CL #cs.LG
paper · pdf · doi:10.18653/v1/2021.acl-long.379
ACL-IJCNLP 2021
openalex publication_date 2021/01/01 · openalex created_date 2021/07/05 · arxiv created 2022/03/03 · arxiv updated 2022/03/04 · openalex updated_date 2026/08/05
Human language understanding operates at multiple levels of granularity (e.g., words, phrases, and sentences) with increasing levels of abstraction that can be hierarchically combined. However, existing deep models with stacked layers do not explicitly model any sort of hierarchical process. This paper proposes a recursive Transformer model based on differentiable CKY style binary trees to emulate the composition process. We extend the bidirectional language model pre-training objective to this architecture, attempting to predict each word given its left and right abstraction nodes. To scale up our approach, we also introduce an efficient pruned tree induction algorithm to enable encoding in just a linear number of composition steps. Experimental results on language modeling and unsupervised parsing show the effectiveness of our approach.