BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model

2019/02/11 by Alex Wang, Kyunghyun Cho, Wang, Alex +1 · 2 voices · 9 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling #cs.CL #cs.LG

paper · pdf · doi:10.48550/arxiv.1902.04094

openalex publication_date 2019/02/11 · arxiv published 2019/02/11 · arxiv updated 2019/04/09 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We show that BERT (Devlin et al., 2018) is a Markov random field language model. This formulation gives way to a natural procedure to sample sentences from BERT. We generate from BERT and find that it can produce high-quality, fluent generations. Compared to the generations of a traditional left-to-right language model, BERT generates sentences that are more diverse but of slightly worse quality.

Citations

• Pointer Sentinel Mixture Models
• Cross-lingual Language Model Pretraining
• Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books

Cited by

• Dual-objective Language Models: Training Efficiency Without Overfitting
• From Interface to Inference: Eliciting Any-Order Inference from Any-Order Models
• MultiFixer: A Coordinator-Proposer Based Multi-Agent Framework For Fixing Multi-Hunk Bugs
• The Hidden Cost of Modeling P(X): Vulnerability to Membership Inference Attacks in Generative Text Classifiers
• HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training
• Reward-Weighted Sampling: Enhancing Non-Autoregressive Characteristics in Masked Diffusion LLMs
• Protein Language Model Fitness Is a Matter of Preference
• On Language Models for Creoles
• L2RS: A Learning-to-Rescore Mechanism for Automatic Speech Recognition

Discussions

• Bert Has a Mouth, and It Must Speak:Bert as a Markov Random Field Language Model [hn, 3 points, 1 comments]
• There are ways around this, eg arxiv.org/abs/1902.04094, but I think uniform rate MLMs are the cleaner approach. Plus, as shown in Eq 5 of the Factorization Curse paper, it's what you get after margin [bsky, 1 points, 0 comments]

Related

• Assessing BERT's Syntactic Abilities
• What the [MASK]? Making Sense of Language-Specific BERT Models
• A Primer in BERTology: What we know about how BERT works
• Investigating BERT's Knowledge of Language: Five Analysis Methods with\n NPIs
• What does BERT learn about prosody?