2014/08/21 by Daniel Godfrey, Godfrey, Daniel, Caley Johns +8 · 46 citations
Computer Science · Mathematics · Physics and Astronomy · #Complex Network Analysis Techniques #Computation and Language (cs.CL) #Computer science #Data mining #Data science #FOS: Computer and information sciences #H.2.8 #H.3.3 #I.2.7 #I.5.4 #Information Retrieval (cs.IR) #Information retrieval #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Natural language processing #Sentiment Analysis and Opinion Mining #Sentiment analysis #Social media #Web Data Mining and Analysis #World Wide Web #cs.CL #cs.IR #cs.LG #stat.ML
paper · pdf · doi:10.48550/arxiv.1408.5427
published in arXiv (Cornell University) (Cornell University)
arxiv created 2014/08/21 · openalex publication_date 2014/08/21 · arxiv updated 2014/08/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Cluster analysis is a field of data analysis that extracts underlying patterns in data. One application of cluster analysis is in text-mining, the analysis of large collections of text to find similarities between documents. We used a collection of about 30,000 tweets extracted from Twitter just before the World Cup started. A common problem with real world text data is the presence of linguistic noise. In our case it would be extraneous tweets that are unrelated to dominant themes. To combat this problem, we created an algorithm that combined the DBSCAN algorithm and a consensus matrix. This way we are left with the tweets that are related to those dominant themes. We then used cluster analysis to find those topics that the tweets describe. We clustered the tweets using k-means, a commonly used clustering algorithm, and Non-Negative Matrix Factorization (NMF) and compared the results. The two algorithms gave similar results, but NMF proved to be faster and provided more easily interpreted results. We explored our results using two visualization tools, Gephi and Wordle.