Solving Historical Dictionary Codes with a Neural Language Model

2020/10/09 by Christopher Chu, Chu, Christopher, Raphael Valenti +3
Computer Science · #Authorship Attribution and Profiling #Computation and Language (cs.CL) #FOS: Computer and information sciences #Handwritten Text Recognition Techniques #Natural Language Processing Techniques #cs.CL

paper · pdf · doi:10.48550/arxiv.2010.04746

10 pages, 6 figures. To appear in EMNLP 2020

arxiv created 2020/10/09 · openalex publication_date 2020/10/09 · arxiv updated 2020/10/13 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We solve difficult word-based substitution codes by constructing a decoding lattice and searching that lattice with a neural language model. We apply our method to a set of enciphered letters exchanged between US Army General James Wilkinson and agents of the Spanish Crown in the late 1700s and early 1800s, obtained from the US Library of Congress. We are able to decipher 75.1% of the cipher-word tokens correctly.

Citations

• Progress and Tradeoffs in Neural Language Models

Related

• Can Sequence-to-Sequence Models Crack Substitution Ciphers?
• ALICE: An Interpretable Neural Architecture for Generalization in Substitution Ciphers
• Neural Language Models for Nineteenth-Century English
• Neural Decipherment via Minimum-Cost Flow: from Ugaritic to Linear B
• Segmenting Numerical Substitution Ciphers