Deriving Language Models from Masked Language Models

2023/05/24 by Lucas Torroba Hennigen, Yoon Kim, Hennigen, Lucas Torroba +1 · 1 citation
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech and dialogue systems #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2305.15501

openalex publication_date 2023/05/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Masked language models (MLM) do not explicitly define a distribution over language, i.e., they are not language models per se. However, recent work has implicitly treated them as such for the purposes of generation and scoring. This paper studies methods for deriving explicit joint distributions from MLMs, focusing on distributions over two tokens, which makes it possible to calculate exact distributional properties. We find that an approach based on identifying joints whose conditionals are closest to those of the MLM works well and outperforms existing Markov random field-based approaches. We further find that this derived model's conditionals can even occasionally outperform the original MLM's conditionals.

Cited by

• Learning Extrapolative Sequence Transformations from Markov Chains

Related

• Inconsistencies in Masked Language Models
• Exposing the Implicit Energy Networks behind Masked Language Models via Metropolis--Hastings
• Distribution Prompting: Understanding the Expressivity of Language Models Through the Next-Token Distributions They Can Produce
• The Distributional Hypothesis Does Not Fully Explain the Benefits of Masked Language Model Pretraining
• Masked Language Modeling Becomes Conditional Density Estimation for Tabular Data Synthesis