2016/01/19 by Justin F. Brunelle, Michele C. Weigle, Brunelle, Justin F. +3
Computer Science · #Digital Libraries (cs.DL) #FOS: Computer and information sciences #Software Engineering Research #Software Testing and Debugging Techniques #Web Data Mining and Analysis
paper · pdf · doi:10.48550/arxiv.1601.05142
openalex publication_date 2016/01/19 · openalex created_date 2022/10/04 · openalex updated_date 2026/07/28
The web is today's primary publication medium, making web archiving an\nimportant activity for historical and analytical purposes. Web pages are\nincreasingly interactive, resulting in pages that are increasingly difficult to\narchive. Client-side technologies (e.g., JavaScript) enable interactions that\ncan potentially change the client-side state of a representation. We refer to\nrepresentations that load embedded resources via JavaScript as deferred\nrepresentations. It is difficult to archive all of the resources in deferred\nrepresentations and the result is archives with web pages that are either\nincomplete or that erroneously load embedded resources from the live web.\n We propose a method of discovering and crawling deferred representations and\ntheir descendants (representation states that are only reachable through\nclient-side events). We adapt the Dincturk et al. Hypercube model to construct\na model for archiving descendants, and we measure the number of descendants and\nrequisite embedded resources discovered in a proof-of-concept crawl. Our\napproach identified an average of 38.5 descendants per seed URI crawled, 70.9%\nof which are reached through an onclick event. This approach also added 15.6\ntimes more embedded resources than Heritrix to the crawl frontier, but at a\nrate that was 38.9 times slower than simply using Heritrix. We show that our\ndataset has two levels of descendants. We conclude with proposed crawl policies\nand an analysis of the storage requirements for archiving descendants.\n