arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1409.6182cs.IR
收藏

用于模板检测和内容提取的基准套件

A Benchmark Suite for Template Detection and Content Extraction

Julián Alarte, Josep Silva

摘要:

模板检测和内容提取是Web信息检索的两个主要领域。它们对网页的结构和内容进行不同的分析,以提取文档的某些部分。然而,他们的目标是不同的。模板检测识别网页的模板(通常与同一网站的其他网页进行比较),内容提取识别网页的主要内容,丢弃其他部分。因此,它们在某种程度上是互补的,因为主要内容不是模板的一部分。据测量,模板代表了Web上40%到50%的数据。因此,识别模板对于索引任务至关重要,因为模板通常包含不相关的信息,如广告,菜单和横幅。处理和存储这些信息很可能导致资源(存储空间、带宽等)的浪费。类似地,识别主要内容对于许多信息检索任务是必不可少的。在本文中,我们提出了一个基准测试套件来测试不同的模板检测和内容提取方法。该套件是公开的,它包含真正的异构网页,这些网页已经被标记,以便不同的技术可以适合(和自动)进行比较。

英文摘要:

Template detection and content extraction are two of the main areas of information retrieval applied to the Web. They perform different analyses over the structure and content of webpages to extract some part of the document. However, their objective is different. While template detection identifies the template of a webpage (usually comparing with other webpages of the same website), content extraction identifies the main content of the webpage discarding the other part. Therefore, they are somehow complementary, because the main content is not part of the template. It has been measured that templates represent between 40% and 50% of data on the Web. Therefore, identifying templates is essential for indexing tasks because templates usually contain irrelevant information such as advertisements, menus and banners. Processing and storing this information is likely to lead to a waste of resources (storage space, bandwidth, etc.). Similarly, identifying the main content is essential for many information retrieval tasks. In this paper, we present a benchmark suite to test different approaches for template detection and content extraction. The suite is public, and it contains real heterogeneous webpages that have been labelled so that different techniques can be suitable (and automatically) compared.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑