arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2112.06540cs.IRcs.CL
收藏

ColBERT的令牌剪枝研究

A Study on Token Pruning for ColBERT

Naver Labs Europe(Naver Labs 欧洲);Naver Corp(Naver 公司)

Carlos Lassance, Maroua Maachou, Joohee Park, Stéphane Clinchant

摘要:

ColBERT模型最近被提出作为一种有效的基于BERT的排名器。通过采用后期交互机制,ColBERT的一个主要优点是可以预先计算文档表示。然而,该模型的最大缺点是索引大小,它与集合中的令牌数量呈线性关系。在本文中,我们研究了各种设计ColBERT模型,以解决这个问题。虽然压缩技术已被探索,以减少索引的大小,在本文中,我们研究标记修剪技术的ColBERT。我们比较简单的算法,以及一个单层的注意力机制,以选择标记保持在索引时间。我们的实验表明,ColBERT索引可以修剪高达30%的MS MARCO通道收集没有显着下降的性能。最后,我们在MS MARCO文档上进行了实验,揭示了这种机制面临的一些挑战。

英文摘要:

The ColBERT model has recently been proposed as an effective BERT based ranker. By adopting a late interaction mechanism, a major advantage of ColBERT is that document representations can be precomputed in advance. However, the big downside of the model is the index size, which scales linearly with the number of tokens in the collection. In this paper, we study various designs for ColBERT models in order to attack this problem. While compression techniques have been explored to reduce the index size, in this paper we study token pruning techniques for ColBERT. We compare simple heuristics, as well as a single layer of attention mechanism to select the tokens to keep at indexing time. Our experiments show that ColBERT indexes can be pruned up to 30\% on the MS MARCO passage collection without a significant drop in performance. Finally, we experiment on MS MARCO documents, which reveal several challenges for such mechanism.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑