arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1605.09362cs.IR
收藏

基于重复字符串集合的文献检索

Document Retrieval on Repetitive String Collections

Travis Gagie, Aleksi Hartikainen, Kalle Karhu, Juha Kärkkäinen, Gonzalo Navarro, Simon J. Puglisi, Jouni Sirén

摘要:

当今增长最快的字符串集合大多是重复的,也就是说,大多数组成文档与许多其他文档相似。随着这些集合的不断增长,处理它们的一个关键方法是利用它们的重复性,这可以将它们的空间使用减少几个数量级。我们研究的问题,索引重复的字符串集合,以执行有效的文件检索操作。文档检索问题通常由搜索引擎在大的自然语言集合上解决,但在通用字符串集合上的技术发展较少。重复字符串集合的情况更不被理解,并且现有的解决方案很少。我们开发了两个新的想法,{ \em交织的LCP}和{ \em预计算的文档列表},产生高度压缩的索引解决的问题,文档列表(找到所有的文件中出现的字符串),顶部- $k$文档检索(找到$k$文档中出现的字符串最频繁),和文档计数(计算的文件中出现的字符串的数量)。我们还表明,一个经典的数据结构,支持后者的查询变得高度可压缩的重复数据。最后,我们展示了如何结合我们开发的工具来解决简单的tf-idf相关性模型下的排名合取和析取多项查询。我们在各种现实生活中的重复性场景中彻底评估所产生的技术,并为每种情况推荐最佳选择。

英文摘要:

Most of the fastest-growing string collections today are repetitive, that is, most of the constituent documents are similar to many others. As these collections keep growing, a key approach to handling them is to exploit their repetitiveness, which can reduce their space usage by orders of magnitude. We study the problem of indexing repetitive string collections in order to perform efficient document retrieval operations on them. Document retrieval problems are routinely solved by search engines on large natural language collections, but the techniques are less developed on generic string collections. The case of repetitive string collections is even less understood, and there are very few existing solutions. We develop two novel ideas, {\em interleaved LCPs} and {\em precomputed document lists}, that yield highly compressed indexes solving the problem of document listing (find all the documents where a string appears), top-$k$ document retrieval (find the $k$ documents where a string appears most often), and document counting (count the number of documents where a string appears). We also show that a classical data structure supporting the latter query becomes highly compressible on repetitive data. Finally, we show how the tools we developed can be combined to solve ranked conjunctive and disjunctive multi-term queries under the simple tf-idf model of relevance. We thoroughly evaluate the resulting techniques in various real-life repetitiveness scenarios, and recommend the best choices for each case.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑