arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2007.08709cs.IR
收藏

计算术语共现频率的可扩展方法

Scalable Methods for Calculating Term Co-Occurrence Frequencies

Bodo Billerbeck, Justin Zobel, Nicholas Lester, Nick Craswell

摘要:

搜索技术利用诸如词频和文档长度的基本信息来计算相似性权重。它们还可以利用更丰富的统计数据,特别是任何两个术语同时出现的文档数量。在本文中,我们提出了计算此统计量的替代方法,这是一项具有挑战性的任务,因为不同的术语对的数量是巨大的-例如,在典型的1000字新闻文章中约有10万个。相反,我们不采用近似算法,因为我们希望能够找到精确的计数。我们探索了它们的效率,发现基于字典的简单方法确实非常慢,而基于倒排索引和线性扫描组合的方法提供了大规模的加速和更好的渐近行为。我们仔细的实现表明,使用我们新颖的列表对方法,每小时可以处理数十万个文档。

英文摘要:

Search techniques make use of elementary information such as term frequencies and document lengths in computation of similarity weighting. They can also exploit richer statistics, in particular the number of documents in which any two terms co-occur. In this paper we propose alternative methods for computing this statistic, a challenging task because the number of distinct pairs of terms is vast -- around 100,000 in a typical 1000-word news article, for example. In contrast, we do not employ approximation algorithms, as we want to be able to find exact counts. We explore their efficiency, finding that a naïve approach based on a dictionary is indeed very slow, while methods based on a combination of inverted indexes and linear scanning provide both massive speed-ups and better observed asymptotic behaviour. Our careful implementation shows that, with our novel list-pairs approach it is possible to process over several hundred thousand documents per hour.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑