面向无推理学习型稀疏检索器的竞争力搜索相关性
Towards Competitive Search Relevance For Inference-Free Learned Sparse Retrievers
AI总结：
本文针对无推理稀疏检索器搜索相关性不足的问题，提出IDF感知惩罚与异构集成知识蒸馏框架，在BEIR基准上超越SOTA模型3.3 NDCG@10，实现与siamese稀疏检索器相当的相关性且延迟仅为BM25的1.1倍。
摘要：
学习型稀疏检索能够通过成熟的倒排索引引擎高效执行检索，近年来获得了越来越多的关注。特别是无推理稀疏检索器极具吸引力，因为它们消除了检索阶段的在线模型推理，从而避免了巨大的计算成本，提供了合理的吞吐量和延迟。然而，即使是SOTA无推理稀疏模型，在搜索相关性方面也远远落后于稀疏和稠密siamese模型。为了实现无推理稀疏检索器具有竞争力的搜索相关性，我们认为它们应该配备专门的训练方法，而不是使用与siamese编码器相同的方法。在本文中，我们提出了两种不同的方法来提升性能。首先，我们为匹配函数提出了一种IDF感知惩罚，以抑制低IDF token的贡献，并增加模型对信息性术语的关注。此外，我们提出了一个异构集成知识蒸馏框架，结合siamese稠密和稀疏检索器在预训练阶段生成监督信号。稠密和稀疏检索器的集成框架利用了它们各自的优势，为知识蒸馏提供了强大的上界。为了应对来自异构监督者的多样化反馈，我们对teacher模型的输出进行归一化然后聚合，以消除分数尺度差异。在BEIR基准上，我们的模型比现有的SOTA无推理稀疏模型高出3.3 NDCG@10分数。它展现出与siamese稀疏检索器相当的搜索相关性，且客户端延迟仅为BM25的1.1倍。
英文摘要：
Learned sparse retrieval, which can efficiently perform retrieval through mature inverted-index engines, has garnered growing attention in recent years. Particularly, the inference-free sparse retrievers are attractive as they eliminate online model inference in the retrieval phase thereby avoids huge computational cost, offering reasonable throughput and latency. However, even the state-of-the-art (SOTA) inference-free sparse models lag far behind in terms of search relevance when compared to both sparse and dense siamese models. Towards competitive search relevance for inference-free sparse retrievers, we argue that they deserve dedicated training methods other than using same ones with siamese encoders. In this paper, we propose two different approaches for performance improvement. First, we propose an IDF-aware penalty for the matching function that suppresses the contribution of low-IDF tokens and increases the model's focus on informative terms. Moreover, we propose a heterogeneous ensemble knowledge distillation framework that combines siamese dense and sparse retrievers to generate supervisory signals during the pre-training phase. The ensemble framework of dense and sparse retriever capitalizes on their strengths respectively, providing a strong upper bound for knowledge distillation. To concur the diverse feedback from heterogeneous supervisors, we normalize and then aggregate the outputs of the teacher models to eliminate score scale differences. On the BEIR benchmark, our model outperforms existing SOTA inference-free sparse model by \textbf{3.3 NDCG@10 score}. It exhibits search relevance comparable to siamese sparse retrievers and client-side latency only \textbf{1.1x that of BM25}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
