arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.13291cs.IR
收藏

晚期交互模型的匹配机制与Token剪枝分析

An Analysis on Matching Mechanisms and Token Pruning for Late-interaction Models

Qi Liu, Gang Guo, Jiaxin Mao, Zhicheng Dou, Ji-Rong Wen, Hao Jiang, Xinyu Zhang, Zhao Cao

AI总结:

本文分析了ColBERT和COIL等晚期交互模型的匹配机制,发现其max求和操作依赖共现信号和重要词,据此提出文档与查询剪枝方法,在保持检索效果的同时显著降低存储开销和延迟。

摘要:

随着预训练语言模型的发展,密集检索模型已成为依赖精确匹配和稀疏词袋表示的传统检索模型的有前途的替代方案。与大多数使用双编码器将每个查询或文档编码为密集向量的密集检索模型不同,最近提出的晚期交互多向量模型(即ColBERT和COIL)通过使用所有token嵌入来表示文档和查询,并使用max求和操作对其相关性进行建模,从而实现了最先进的检索效果。然而,这些细粒度的表示可能会给实际的搜索系统带来不可接受的存储开销。在本研究中,我们系统地分析了这些晚期交互模型的匹配机制,并表明max求和操作严重依赖于共现信号和文档中的一些重要单词。基于这些发现,我们随后提出了几种简单的文档剪枝方法来减少存储开销,并比较了不同剪枝方法在不同晚期交互模型上的有效性。我们还利用查询剪枝方法来进一步减少检索延迟。我们在域内和域外数据集上进行了广泛的实验,结果表明,所使用的一些剪枝方法可以显著提高这些晚期交互模型的效率,而不会实质性地损害其检索效果。

英文摘要:

With the development of pre-trained language models, the dense retrieval models have become promising alternatives to the traditional retrieval models that rely on exact match and sparse bag-of-words representations. Different from most dense retrieval models using a bi-encoder to encode each query or document into a dense vector, the recently proposed late-interaction multi-vector models (i.e., ColBERT and COIL) achieve state-of-the-art retrieval effectiveness by using all token embeddings to represent documents and queries and modeling their relevance with a sum-of-max operation. However, these fine-grained representations may cause unacceptable storage overhead for practical search systems. In this study, we systematically analyze the matching mechanism of these late-interaction models and show that the sum-of-max operation heavily relies on the co-occurrence signals and some important words in the document. Based on these findings, we then propose several simple document pruning methods to reduce the storage overhead and compare the effectiveness of different pruning methods on different late-interaction models. We also leverage query pruning methods to further reduce the retrieval latency. We conduct extensive experiments on both in-domain and out-domain datasets and show that some of the used pruning methods can significantly improve the efficiency of these late-interaction models without substantially hurting their retrieval effectiveness.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑