2016/11/24 by Daniel Gayo-Avello, Gayo-Avello, Daniel
Computer Science · Physics and Astronomy · Social Sciences · #Advanced Text Analysis Techniques #Complex Network Analysis Techniques #Computers and Society (cs.CY) #Digital Libraries (cs.DL) #FOS: Computer and information sciences #Radio, Podcasts, and Digital Media #Social and Information Networks (cs.SI) #Video Analysis and Summarization #Web Data Mining and Analysis #cs.CY #cs.DL #cs.SI
paper · pdf · doi:10.48550/arxiv.1611.08144
22 pages, 13 figures
arxiv created 2016/11/24 · openalex publication_date 2016/11/24 · arxiv updated 2016/11/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Twitter is among the commonest sources of data employed in social media research mainly because of its convenient APIs to collect tweets. However, most researchers do not have access to the expensive Firehose and Twitter Historical Archive, and they must rely on data collected with free APIs whose representativeness has been questioned. In 2010 the Library of Congress announced an agreement with Twitter to provide researchers access to the whole Twitter Archive. However, such a task proved to be daunting and, at the moment of this writing, no researcher has had the opportunity to access such materials. Still, there have been experiences that proved that smaller searchable archives are feasible and, therefore, amenable for academics to build with relatively little resources. In this paper I describe my efforts to build one of such archives, covering the first three years of Twitter (actually from March 2006 to July 2009) and containing 1.48 billion tweets. If you carefully follow my directions you may have your very own little Twitter Historical Archive and you may forget about paying for historical tweets. Please note that to achieve that you should be proficient in some programming language, knowable about Twitter APIs, and have some basic knowledge on ElasticSearch; moreover, you may very well get disappointed by the quality of the contents of the final dataset.