arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1902.09667cs.IR
收藏

在Web上引导特定于域的内容发现

Bootstrapping Domain-Specific Content Discovery on the Web

Kien Pham, Aécio Santos, Juliana Freire

摘要:

从Web上持续发现特定于域的内容的能力对于许多应用程序来说是至关重要的。虽然集中的爬行策略已被证明是有效的发现,配置一个集中的爬虫是困难和耗时的。给定一个感兴趣的域$D$,主题专家(SME)必须搜索相关网站并收集一组代表性的Web页面,以用作用于创建识别$D$中的页面的分类器的训练样本,以及一组用于种子抓取的页面。在本文中,我们提出了DISCO,旨在引导特定领域的搜索方法。给定一小部分网站,DISCO的目标是发现大量相关网站。DISCO使用一个基于排名的框架,模仿用户在Web上搜索信息的方式:它迭代地发现新页面,提取并对它们进行排名。它还应用多种发现策略,包括向搜索引擎发出的基于关键字的和相关的查询、向后和向前爬行。通过系统地结合这些战略,DISCO能够在各种领域实现高收获率和覆盖率。我们在四个社会良好的领域进行了广泛的实验,使用中小企业在各自的领域收集的数据,并表明我们的方法是有效的,优于国家的最先进的方法。

英文摘要:

The ability to continuously discover domain-specific content from the Web is critical for many applications. While focused crawling strategies have been shown to be effective for discovery, configuring a focused crawler is difficult and time-consuming. Given a domain of interest $D$, subject-matter experts (SMEs) must search for relevant websites and collect a set of representative Web pages to serve as training examples for creating a classifier that recognizes pages in $D$, as well as a set of pages to seed the crawl. In this paper, we propose DISCO, an approach designed to bootstrap domain-specific search. Given a small set of websites, DISCO aims to discover a large collection of relevant websites. DISCO uses a ranking-based framework that mimics the way users search for information on the Web: it iteratively discovers new pages, distills, and ranks them. It also applies multiple discovery strategies, including keyword-based and related queries issued to search engines, backward and forward crawling. By systematically combining these strategies, DISCO is able to attain high harvest rates and coverage for a variety of domains. We perform extensive experiments in four social-good domains, using data gathered by SMEs in the respective domains, and show that our approach is effective and outperforms state-of-the-art methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑