2003/01/01 by Egidio L. Terra, Charles L. A. Clarke · 1 citation
Computer Science · #Advanced Text Analysis Techniques #Artificial intelligence #Computer science #Context (archaeology) #Information retrieval #Linguistics #Natural Language Processing Techniques #Natural language processing #Semantic similarity #Sentence #Set (abstract data type) #Similarity (geometry) #Synonym (taxonomy) #Topic Modeling #Word (group theory) #Word lists by frequency #Word2vec
paper · pdf · doi:10.3115/1073445.1073477
openalex publication_date 2003/01/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/29
Statistical measures of word similarity have application in many areas of natural language processing, such as language modeling and information retrieval. We report a comparative study of two methods for estimating word co-occurrence frequencies required by word similarity measures. Our frequency estimates are generated from a terabyte-sized corpus of Web data, and we study the impact of corpus size on the effectiveness of the measures. We base the evaluation on one TOEFL question set and two practice questions sets, each consisting of a number of multiple choice questions seeking the best synonym for a given target word. For two question sets, a context for the target word is provided, and we examine a number of word similarity measures that exploit this context. Our best combination of similarity measure and frequency estimation method answers 6-8% more questions than the best results previously reported for the same question sets.