1994/12/06 by J.P. Ueberla, Joerg P. Ueberla, Ueberla, Joerg P. · 1 citation
Computer Science · Physics and Astronomy · #Bayesian Methods and Mixture Models #Computation and Language (cs.CL) #Data Management and Algorithms #FOS: Computer and information sciences #Opinion Dynamics and Social Influence #cmp-lg #cs.CL
paper · pdf · doi:10.48550/arxiv.cmp-lg/9412003
27 pages, latex, comments welcome
arxiv created 1994/12/06 · openalex publication_date 1994/12/06 · arxiv updated 2009/11/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Statistical language models frequently suffer from a lack of training data. This problem can be alleviated by clustering, because it reduces the number of free parameters that need to be trained. However, clustered models have the following drawback: if there is ``enough'' data to train an unclustered model, then the clustered variant may perform worse. On currently used language modeling corpora, e.g. the Wall Street Journal corpus, how do the performances of a clustered and an unclustered model compare? While trying to address this question, we develop the following two ideas. First, to get a clustering algorithm with potentially high performance, an existing algorithm is extended to deal with higher order N-grams. Second, to make it possible to cluster large amounts of training data more efficiently, a heuristic to speed up the algorithm is presented. The resulting clustering algorithm can be used to cluster trigrams on the Wall Street Journal corpus and the language models it produces can compete with existing back-off models. Especially when there is only little training data available, the clustered models clearly outperform the back-off models.