Learning to Create and Reuse Words in Open-Vocabulary Neural Language Modeling

2017/04/23 by Kazuya Kawakami, Chris Dyer, Kawakami, Kazuya +3 · 1 citation
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1704.06986

openalex publication_date 2017/04/23 · openalex created_date 2017/05/05 · openalex updated_date 2026/07/28

Abstract

Fixed-vocabulary language models fail to account for one of the most characteristic statistical facts of natural language: the frequent creation and reuse of new word types. Although character-level language models offer a partial solution in that they can create word types not attested in the training corpus, they do not capture the "bursty" distribution of such words. In this paper, we augment a hierarchical LSTM language model that generates sequences of word tokens character by character with a caching mechanism that learns to reuse previously generated words. To validate our model we construct a new open-vocabulary language modeling corpus (the Multilingual Wikipedia Corpus, MWC) from comparable Wikipedia articles in 7 typologically diverse languages and demonstrate the effectiveness of our model across this range of languages.

Citations

• Recurrent Dropout Without Memory Loss
• Hierarchical Multiscale Recurrent Neural Networks
• Neural Architecture Search with Reinforcement Learning

Cited by

• Handling Open-Vocabulary Constructs in Formalizing Specifications: Retrieval-Augmented Parsing with Expert Knowledge

Related

• Gated Word-Character Recurrent Language Model
• From Characters to Words: Hierarchical Pre-trained Language Model for Open-vocabulary Language Understanding
• Spell Once, Summon Anywhere: A Two-Level Open-Vocabulary Language Model
• Language Models with Pre-Trained (GloVe) Word Embeddings
• Syntax-Aware Language Modeling with Recurrent Neural Networks