CoCon: A Self-Supervised Approach for Controlled Text Generation

2020/06/05 by Alvin Chan, Chan, Alvin, Yew-Soon Ong +8 · 16 citations
Computer Science · Mathematics · #Artificial intelligence #Computation and Language (cs.CL) #Computer science #Content (measure theory) #Control (management) #FOS: Computer and information sciences #Language model #Machine Learning (cs.LG) #Mathematics #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Natural language processing #Neural and Evolutionary Computing (cs.NE) #Phrase #Text generation #Topic Modeling #Transformer #cs.CL #cs.LG #cs.NE

paper · pdf · doi:10.48550/arxiv.2006.03535

published in PolyPublie (École Polytechnique de Montréal) (Polytechnique Montréal) · ICLR 2021 Camera-Ready

openalex publication_date 2020/06/05 · arxiv created 2022/06/10 · arxiv updated 2022/06/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/29

Abstract

Pretrained Transformer-based language models (LMs) display remarkable natural language generation capabilities. With their immense potential, controlling text generation of such LMs is getting attention. While there are studies that seek to control high-level attributes (such as sentiment and topic) of generated text, there is still a lack of more precise control over its content at the word- and phrase-level. Here, we propose Content-Conditioner (CoCon) to control an LM's output text with a content input, at a fine-grained level. In our self-supervised approach, the CoCon block learns to help the LM complete a partially-observed text sequence by conditioning with content inputs that are withheld from the LM. Through experiments, we show that CoCon can naturally incorporate target content into generated texts and control high-level text attributes in a zero-shot manner.

Citations

• The Curious Case of Neural Text Degeneration
• CTRL: A Conditional Transformer Language Model for Controllable Generation
• Fine-Tuning Language Models from Human Preferences
• Simple, Scalable Adaptation for Neural Machine Translation
• Synthesizer: Rethinking Self-Attention in Transformer Models

Cited by

• Weak-to-Strong On-Policy Distillation
• DTPA: Dynamic Token-level Prefix Augmentation for Controllable Text Generation
• Plug-and-Blend: A Framework for Controllable Story Generation with Blended Control Codes
• When Does Metadata Conditioning (NOT) Work for Language Model Pre-Training? A Study with Context-Free Grammars

Related

• Plug and Play Language Models: A Simple Approach to Controlled Text Generation
• A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models
• Controlled Text Generation with Hidden Representation Transformations
• Conditioned Natural Language Generation using only Unconditioned Language Model: An Exploration
• Self-conditioning pre-trained language models