2019/02/25 by Kien Pham, Aécio Santos, Aecio Santos +1
Computer Science · #Bootstrapping (finance) #Classifier (UML) #Crawling #Information Retrieval and Search Behavior #Search engine #Set (abstract data type) #Topic Modeling #Variety (cybernetics) #Web Data Mining and Analysis #Web crawler #Web page #cs.IR
paper · pdf · doi:10.1145/3308558.3313709
Accepted for publication in the Proceedings of the 2019 World Wide Web Conference (WWW'19). 11 pages, 8 figures
openalex created_date 2019/02/21 · arxiv created 2019/02/25 · arxiv updated 2019/02/27 · openalex publication_date 2019/05/13 · openalex updated_date 2026/08/05
The ability to continuously discover domain-specific content from the Web is critical for many applications. While focused crawling strategies have been shown to be effective for discovery, configuring a focused crawler is difficult and time-consuming. Given a domain of interest D, subject-matter experts (SMEs) must search for relevant websites and collect a set of representative Web pages to serve as training examples for creating a classifier that recognizes pages in D, as well as a set of pages to seed the crawl. In this paper, we propose DISCO, an approach designed to bootstrap domain-specific search. Given a small set of websites , DISCO aims to discover a large collection of relevant websites . DISCO uses a ranking-based framework that mimics the way users search for information on the Web: it iteratively discovers new pages, distills, and ranks them. It also applies multiple discovery strategies, including keyword-based and related queries issued to search engines, backward and forward crawling. By systematically combining these strategies, DISCO is able to attain high harvest rates and coverage for a variety of domains. We perform extensive experiments in four social-good domains, using data gathered by SMEs in the respective domains, and show that our approach is effective and outperforms state-of-the-art methods.