Efficient Calculation of Bigram Frequencies in a Corpus of Short Texts

2016/04/18 by Melvyn Drag, Drag, Melvyn, Gauthaman Vasudevan +1
Computer Science · #Algorithms and Data Compression #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1604.05559

openalex publication_date 2016/04/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

We show that an efficient and popular method for calculating bigram frequencies is unsuitable for bodies of short texts and offer a simple alternative. Our method has the same computational complexity as the old method and offers an exact count instead of an approximation.

Related

• A Comparison of Two Smoothing Methods for Word Bigram Models
• A New Statistical Parser Based on Bigram Lexical Dependencies
• Language Detection by Means of the Minkowski Norm: Identification Through Character Bigrams and Frequency Analysis
• Introducing DeReKoGram: A Novel Frequency Dataset with Lemma and Part-of-Speech Information for German
• Computing q-gram Non-overlapping Frequencies on SLP Compressed Texts