2012/07/08 by Ted Dunning, Dunning, Ted
Biochemistry, Genetics and Molecular Biology · Computer Science · #Algorithms and Data Compression #Computation and Language (cs.CL) #FOS: Computer and information sciences #Fractal and DNA sequence analysis #Information Retrieval (cs.IR) #Machine Learning in Bioinformatics #cs.CL #cs.IR
paper · pdf · doi:10.48550/arxiv.1207.1847
~ 176 pages, many figures
arxiv created 2012/07/08 · openalex publication_date 2012/07/08 · arxiv updated 2012/07/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
The statistical methods derived and described in this thesis provide new ways to elucidate the structural properties of text and other symbolic sequences. Generically, these methods allow detection of a difference in the frequency of a single feature, the detection of a difference between the frequencies of an ensemble of features and the attribution of the source of a text. These three abstract tasks suffice to solve problems in a wide variety of settings. Furthermore, the techniques described in this thesis can be extended to provide a wide range of additional tests beyond the ones described here. A variety of applications for these methods are examined in detail. These applications are drawn from the area of text analysis and genetic sequence analysis. The textually oriented tasks include finding interesting collocations and cooccurent phrases, language identification, and information retrieval. The biologically oriented tasks include species identification and the discovery of previously unreported long range structure in genes. In the applications reported here where direct comparison is possible, the performance of these new methods substantially exceeds the state of the art. Overall, the methods described here provide new and effective ways to analyse text and other symbolic sequences. Their particular strength is that they deal well with situations where relatively little data are available. Since these methods are abstract in nature, they can be applied in novel situations with relative ease.