2013/05/25 by Ahmed AlSum, Michael L. Nelson, AlSum, Ahmed +1
Computer Science · #Data Management and Algorithms #FOS: Computer and information sciences #Graph Theory and Algorithms #Information Retrieval (cs.IR) #Web Data Mining and Analysis #cs.IR
paper · pdf · doi:10.48550/arxiv.1305.5959
openalex publication_date 2013/05/25 · arxiv created 2013/06/11 · arxiv updated 2013/06/12 · openalex created_date 2019/06/27 · openalex updated_date 2026/07/28
Archiving the web is socially and culturally critical, but presents problems of scale. The Internet Archive's Wayback Machine can replay captured web pages as they existed at a certain point in time, but it has limited ability to provide extensive content and structural metadata about the web graph. While the live web has developed a rich ecosystem of APIs to facilitate web applications (e.g., APIs from Google and Twitter), the web archiving community has not yet broadly implemented this level of access. We present ArcLink, a proof-of-concept system that complements open source Wayback Machine installations by optimizing the construction, storage, and access to the temporal web graph. We divide the web graph construction into four stages (filtering, extraction, storage, and access) and explore optimization for each stage. ArcLink extends the current Web archive interfaces to return content and structural metadata for each URI. We show how this API can be applied to such applications as retrieving inlinks, outlinks, anchortext, and PageRank.