arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1812.09221cs.IR
收藏

维基百科文本重用:内部和外部

Wikipedia Text Reuse: Within and Without

Milad Alshomary, Michael Völske, Tristan Licht, Henning Wachsmuth, Benno Stein, Matthias Hagen, Martin Potthast

摘要:

我们通过编译维基百科内和外的第一个文本重用案例语料库(即,在Common Crawl的样本中重用维基百科文本)。为了发现逐字复制和粘贴之外的重用,我们采用了最先进的文本重用检测技术,首次将其扩展为处理整个维基百科作为分布式检索管道的一部分。我们进一步报告了我们发现的1亿个内部重用案例和160万个外部重用案例的试点分析。维基百科内部的文本重用产生了新的任务,如条目模板归纳,修复由于重用段落的异步编辑引起的不一致而导致的质量缺陷,或补充维基百科的本体。维基百科之外的文本重用为量化维基百科在网络上的影响力这一新兴领域提供了一个有形的衡量标准。为了促进未来对这些任务的研究,并为了可重复性,维基百科文本重用语料库和检索管道是免费提供的。

英文摘要:

We study text reuse related to Wikipedia at scale by compiling the first corpus of text reuse cases within Wikipedia as well as without (i.e., reuse of Wikipedia text in a sample of the Common Crawl). To discover reuse beyond verbatim copy and paste, we employ state-of-the-art text reuse detection technology, scaling it for the first time to process the entire Wikipedia as part of a distributed retrieval pipeline. We further report on a pilot analysis of the 100 million reuse cases inside, and the 1.6 million reuse cases outside Wikipedia that we discovered. Text reuse inside Wikipedia gives rise to new tasks such as article template induction, fixing quality flaws due to inconsistencies arising from asynchronous editing of reused passages, or complementing Wikipedia's ontology. Text reuse outside Wikipedia yields a tangible metric for the emerging field of quantifying Wikipedia's influence on the web. To foster future research into these tasks, and for reproducibility's sake, the Wikipedia text reuse corpus and the retrieval pipeline are made freely available.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑