2016/02/19 by Shawn Jones, Jones, Shawn M., Harihar Shankar +1 · 1 citation
Computer Science · #Advanced Malware Detection Techniques #Digital Libraries (cs.DL) #FOS: Computer and information sciences #H.3.7 #Web Application Security Vulnerabilities #Web Data Mining and Analysis
paper · pdf · doi:10.48550/arxiv.1602.06223
openalex publication_date 2016/02/19 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Text extraction from web pages has many applications, including web crawling optimization and document clustering. Though much has been written about the acquisition of content from live web pages, content acquisition of archived web pages, known as mementos, remains a relatively new enterprise. In the course of conducting a study with almost 700,000 web pages, we encountered issues acquiring mementos and extracting text from them. The acquisition of memento content via HTTP is expected to be a relatively painless exercise, but we have found cases to the contrary. We also find that the parsing of HTML, already known to be problematic, can be more complex when one attempts to extract the text of mementos across many web archives, due to issues involving different memento presentation behaviors, as well as the age of the HTML in their mementos. For the benefit of others acquiring mementos across many web archives, we document those experiences here.