基于多级蒸馏的稠密段落检索模型
A Multi-level Distillation based Dense Passage Retrieval Model
AI总结：
本文提出基于多级蒸馏的稠密检索模型MD2PR，通过句子级和词语级蒸馏将交叉编码器知识转移至双编码器，并引入动态过滤方法，在两个标准数据集上超越11个基线模型。
摘要：
排序器和检索器是稠密段落检索中的两个重要组成部分。检索器通常采用双编码器模型，其中查询和文档分别输入到两个预训练模型中，模型生成的向量用于相似度计算。排序器通常使用交叉编码器模型，其中拼接的查询-文档对输入到预训练模型中以获得词语相似度。然而，双编码器模型由于独立编码而缺乏查询和文档之间的交互，而交叉编码器模型需要大量的计算成本进行注意力计算，难以获得实时检索结果。在本文中，我们提出了一种基于多级蒸馏的稠密检索模型MD2PR。在该模型中，我们在句子级别和词语级别将交叉编码器学到的知识蒸馏到双编码器。句子级蒸馏增强了双编码器捕捉句子主题和情感的能力。词语级蒸馏提高了双编码器在词语语义和关系分析方面的能力。因此，双编码器可以独立用于后续的编码和检索，避免了交叉编码器参与带来的大量计算成本。此外，我们提出了一种简单的动态过滤方法，该方法在多次训练迭代中更新阈值，以确保有效识别假阴性，从而获得更全面的语义表示空间。在两个标准数据集上的实验结果表明，我们的MD2PR在MRR和Recall指标上优于11个基线模型。
英文摘要：
Ranker and retriever are two important components in dense passage retrieval. The retriever typically adopts a dual-encoder model, where queries and documents are separately input into two pre-trained models, and the vectors generated by the models are used for similarity calculation. The ranker often uses a cross-encoder model, where the concatenated query-document pairs are input into a pre-trained model to obtain word similarities. However, the dual-encoder model lacks interaction between queries and documents due to its independent encoding, while the cross-encoder model requires substantial computational cost for attention calculation, making it difficult to obtain real-time retrieval results. In this paper, we propose a dense retrieval model called MD2PR based on multi-level distillation. In this model, we distill the knowledge learned from the cross-encoder to the dual-encoder at both the sentence level and word level. Sentence-level distillation enhances the dual-encoder on capturing the themes and emotions of sentences. Word-level distillation improves the dual-encoder in analysis of word semantics and relationships. As a result, the dual-encoder can be used independently for subsequent encoding and retrieval, avoiding the significant computational cost associated with the participation of the cross-encoder. Furthermore, we propose a simple dynamic filtering method, which updates the threshold during multiple training iterations to ensure the effective identification of false negatives and thus obtains a more comprehensive semantic representation space. The experimental results over two standard datasets show our MD2PR outperforms 11 baseline models in terms of MRR and Recall metrics.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
