2024/06/19 by Jindřich Libovický, Libovický, Jindřich, Jindřich Helcl +1 · 8 citations
Arts and Humanities · Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Lexicography and Language Studies #Natural Language Processing Techniques #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2406.13560
openalex publication_date 2024/06/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
We present three innovations in tokenization and subword segmentation. First, we propose to use unsupervised morphological analysis with Morfessor as pre-tokenization. Second, we present an algebraic method for obtaining subword embeddings grounded in a word embedding space. Based on that, we design a novel subword segmentation algorithm that uses the embeddings, ensuring that the procedure considers lexical meaning. Third, we introduce an efficient segmentation algorithm based on a subword bigram model that can be initialized with the lexically aware segmentation method to avoid using Morfessor and large embedding tables at inference time. We evaluate the proposed approaches using two intrinsic metrics and measure their performance on two downstream tasks: part-of-speech tagging and machine translation. Our experiments show significant improvements in the morphological plausibility of the segmentation when evaluated using segmentation precision on morpheme boundaries and improved Rényi efficiency in 8 languages. Although the proposed tokenization methods do not have a large impact on automatic translation quality, we observe consistent performance gains in the arguably more morphological task of part-of-speech tagging.