Syntax-driven Iterative Expansion Language Models for Controllable Text Generation

2020/04/05 by Noé Casas, Noe Casas, José A. R. Fonollosa +5 · 4 citations
Computer Science · #Algorithm #Artificial intelligence #Computation and Language (cs.CL) #Computer science #Decoding methods #Dependency (UML) #FOS: Computer and information sciences #Language model #Natural Language Processing Techniques #Natural language processing #Parsing #Software Engineering Research #Syntax #Text generation #Topic Modeling #Transformer #Treebank #Voltage #cs.CL

paper · pdf · doi:10.48550/arxiv.2004.02211

published in arXiv (Cornell University) (Cornell University) · Accepted at the EMNLP 2020 Workshop on Structured Prediction for NLP

openalex publication_date 2020/04/05 · arxiv created 2020/10/30 · arxiv updated 2020/11/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05

Abstract

The dominant language modeling paradigm handles text as a sequence of discrete tokens. While that approach can capture the latent structure of the text, it is inherently constrained to sequential dynamics for text generation. We propose a new paradigm for introducing a syntactic inductive bias into neural text generation, where the dependency parse tree is used to drive the Transformer model to generate sentences iteratively. Our experiments show that this paradigm is effective at text generation, with quality between LSTMs and Transformers, and comparable diversity, requiring less than half their decoding steps, and its generation process allows direct control over the syntactic constructions of the generated text, enabling the induction of stylistic variations.

Citations

• A Learning Algorithm for Continually Running Fully Recurrent Neural Networks
• Long Short-Term Memory
• Non-Monotonic Sequential Text Generation
• Insertion Transformer: Flexible Sequence Generation via Insertion Operations
• Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Sequence Modeling with Unconstrained Generation Order

Related

• Transformer-Based Neural Text Generation with Syntactic Guidance
• Explicit Syntactic Guidance for Neural Text Generation
• A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models
• Generative Pretrained Structured Transformers: Unsupervised Syntactic Language Models at Scale
• Sentence-Level Content Planning and Style Specification for Neural Text Generation