1998/06/19 by Judith Hochberg, Judy Hochberg, Clint Scovel +7
Computer Science · Physics and Astronomy · #Algorithms and Data Compression #Bayesian Methods and Mixture Models #Computation and Language (cs.CL) #FOS: Computer and information sciences #Scientific Research and Discoveries #cmp-lg #cs.CL
paper · pdf · doi:10.48550/arxiv.cmp-lg/9806012
8 pages, 5 figures. To appear in Proceedings of WVLC-6
arxiv created 1998/06/19 · openalex publication_date 1998/06/19 · arxiv updated 2009/11/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
This paper describes a method for asking statistical questions about a large text corpus. We exemplify the method by addressing the question, "What percentage of Federal Register documents are real documents, of possible interest to a text researcher or analyst?" We estimate an answer to this question by evaluating 200 documents selected from a corpus of 45,820 Federal Register documents. Stratified sampling is used to reduce the sampling uncertainty of the estimate from over 3100 documents to fewer than 1000. The stratification is based on observed characteristics of real documents, while the sampling procedure incorporates a Bayesian version of Neyman allocation. A possible application of the method is to establish baseline statistics used to estimate recall rates for information retrieval systems.