Dependency-based Mixture Language Models

2022/03/19 by Zhixian Yang, Xiaojun Wan, Yang, Zhixian +1
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2203.10256

openalex publication_date 2022/03/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Various models have been proposed to incorporate knowledge of syntactic structures into neural language models. However, previous works have relied heavily on elaborate components for a specific language model, usually recurrent neural network (RNN), which makes themselves unwieldy in practice to fit into other neural language models, such as Transformer and GPT-2. In this paper, we introduce the Dependency-based Mixture Language Models. In detail, we first train neural language models with a novel dependency modeling objective to learn the probability distribution of future dependent tokens given context. We then formulate the next-token probability by mixing the previous dependency modeling probability distributions with self-attention. Extensive experiments and human evaluations show that our method can be easily and effectively applied to different neural language models while improving neural text generation on various tasks.

Related

• Neural Composition: Learning to Generate from Multiple Models
• Dependency Recurrent Neural Language Models for Sentence Completion
• Contextually Structured Token Dependency Encoding for Large Language Models
• Generalizing and Hybridizing Count-based and Neural Language Models
• Dependency Grammar Induction with a Neural Variational Transition-based Parser