Metadata Might Make Language Models Better

2022/11/18 by Beelen, Kaspar, van Strien, Daniel
#Computation and Language (cs.CL) #Digital Libraries (cs.DL) #FOS: Computer and information sciences

paper · doi:10.48550/arxiv.2211.10086

Abstract

This paper discusses the benefits of including metadata when training language models on historical collections. Using 19th-century newspapers as a case study, we extend the time-masking approach proposed by Rosin et al., 2022 and compare different strategies for inserting temporal, political and geographical information into a Masked Language Model. After fine-tuning several DistilBERT on enhanced input data, we provide a systematic evaluation of these models on a set of evaluation tasks: pseudo-perplexity, metadata mask-filling and supervised classification. We find that showing relevant metadata to a language model has a beneficial impact and may even produce more robust and fairer models.

Related

• Metadata Enrichment of Long Text Documents using Large Language Models
• Pretraining Language Models on Historical Text
• Metadata Conditioning Accelerates Language Model Pre-training
• If the Sources Could Talk: Evaluating Large Language Models for Research Assistance in History
• Go Forth and Prosper: Language Modeling with Ancient Textual History