Mimic and Conquer: Heterogeneous Tree Structure Distillation for Syntactic NLP

2020/09/16 by Hao Fei, Fei, Hao, Yafeng Ren +3 · 3 citations
Computer Science · Mathematics · #Algorithm #Artificial intelligence #Artificial neural network #Computation and Language (cs.CL) #Computer science #Data structure #Encoder #FOS: Computer and information sciences #Machine learning #Mathematics #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Natural language processing #Programming language #Propagation of uncertainty #Recurrent neural network #Recursion (computer science) #Simple (philosophy) #Syntax #Topic Modeling #Tree (set theory) #Tree structure #Trie #cs.CL

paper · pdf · doi:10.48550/arxiv.2009.07411

published in arXiv (Cornell University) (Cornell University) · To appear at EMNLP2020

arxiv created 2020/09/16 · openalex publication_date 2020/09/16 · arxiv updated 2020/09/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

Syntax has been shown useful for various NLP tasks, while existing work mostly encodes singleton syntactic tree using one hierarchical neural network. In this paper, we investigate a simple and effective method, Knowledge Distillation, to integrate heterogeneous structure knowledge into a unified sequential LSTM encoder. Experimental results on four typical syntax-dependent tasks show that our method outperforms tree encoders by effectively integrating rich heterogeneous structure syntax, meanwhile reducing error propagation, and also outperforms ensemble methods, in terms of both the efficiency and accuracy.

Citations

• Distilling the Knowledge in a Neural Network
• Recognition and parsing of context-free languages in time n3
• Long Short-Term Memory
• On the Properties of Neural Machine Translation: Encoder-Decoder Approaches
• Deep Learning with Dynamic Computation Graphs
• What you can cram into a single vector: Probing sentence embeddings for linguistic properties
• Constituency Parsing with a Self-Attentive Encoder
• Born Again Neural Networks
• Tree-Structured Neural Machine for Linguistics-Aware Sentence Generation
• Three Generative, Lexicalised Models for Statistical Parsing
• Graph Convolution over Pruned Dependency Trees Improves Relation Extraction
• Head-Driven Phrase Structure Grammar Parsing on Penn Treebank
• A large annotated corpus for learning natural language inference
• Technical report on Conversational Question Answering

Related

• Scalable Syntax-Aware Language Models Using Knowledge Distillation
• Syntactic Structure Distillation Pretraining For Bidirectional Encoders
• Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding
• Bidirectional Tree-Structured LSTM with Head Lexicalization
• Distillation of encoder-decoder transformers for sequence labelling