arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1801.02607cs.IR
收藏

Web2Text:深度结构化样板文件删除

Web2Text: Deep Structured Boilerplate Removal

Thijs Vogels, Octavian-Eugen Ganea, Carsten Eickhoff

摘要:

网页是许多自然语言处理和信息检索任务的有价值的信息源。从这些文档中提取主要内容对于派生应用程序的性能至关重要。为了解决这个问题,我们引入了一个新的模型,执行序列标签,集体分类HTML页面中的所有文本块作为样板或主要内容。我们的方法使用了一个隐藏的马尔可夫模型上的潜力来自DOM树的功能,使用卷积神经网络。该方法在CleanEval基准测试中为样板文件的删除设置了一个新的最先进的性能。作为信息检索管道的一个组件,它提高了在CITCHWeb 12集合上的检索性能。

英文摘要:

Web pages are a valuable source of information for many natural language processing and information retrieval tasks. Extracting the main content from those documents is essential for the performance of derived applications. To address this issue, we introduce a novel model that performs sequence labeling to collectively classify all text blocks in an HTML page as either boilerplate or main content. Our method uses a hidden Markov model on top of potentials derived from DOM tree features using convolutional neural networks. The proposed method sets a new state-of-the-art performance for boilerplate removal on the CleanEval benchmark. As a component of information retrieval pipelines, it improves retrieval performance on the ClueWeb12 collection.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑