arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 1604.08897cs.IRcs.DL
收藏

高重复文档集的通用索引

Universal Indexes for Highly Repetitive Document Collections

Francisco Claude, Antonio Fariña, Miguel A. Martínez-Prieto, Gonzalo Navarro

摘要:

在其他应用中,随着版本化文档的大型存储库的出现,索引高度重复的集合已经成为一个相关的问题。这些集合可能会达到巨大的规模,但大多数是由接近其他副本的文档组成的。传统的索引这些集合的技术未能适当地利用它们的索引来减少空间。我们引入了新的技术来压缩倒排索引,利用这种接近复制的规律。它们基于游程长度、Lempel-Ziv或差分倒排列表的语法压缩,而不是通常的间隙编码。我们表明,在这种高度重复的设置,我们的压缩方法显着减少了与经典技术获得的空间,在价格适中的减速。此外,我们最好的方法是通用的,也就是说,它们不需要知道集合的版本结构,也不需要知道一个清晰的版本结构。在比较中,我们还引入了压缩的自索引。这些是为一般字符串(不仅是自然语言文本)设计的,并以集成的形式表示文本集合加上索引结构(而不是倒排索引)。我们表明,这些技术可以进一步压缩,使用我们的新的倒排索引所需的空间的一小部分。然而,它们慢了几个数量级。

英文摘要:

Indexing highly repetitive collections has become a relevant problem with the emergence of large repositories of versioned documents, among other applications. These collections may reach huge sizes, but are formed mostly of documents that are near-copies of others. Traditional techniques for indexing these collections fail to properly exploit their regularities in order to reduce space. We introduce new techniques for compressing inverted indexes that exploit this near-copy regularity. They are based on run-length, Lempel-Ziv, or grammar compression of the differential inverted lists, instead of the usual practice of gap-encoding them. We show that, in this highly repetitive setting, our compression methods significantly reduce the space obtained with classical techniques, at the price of moderate slowdowns. Moreover, our best methods are universal, that is, they do not need to know the versioning structure of the collection, nor that a clear versioning structure even exists. We also introduce compressed self-indexes in the comparison. These are designed for general strings (not only natural language texts) and represent the text collection plus the index structure (not an inverted index) in integrated form. We show that these techniques can compress much further, using a small fraction of the space required by our new inverted indexes. Yet, they are orders of magnitude slower.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑