从蒸馏到硬负采样：使稀疏神经IR模型更有效
From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective
摘要：
基于密集表示与近似最近邻搜索相结合的神经检索器最近受到了很多关注，这是由于它们的成功是为了训练而对示例进行蒸馏和/或更好的采样-同时仍然依赖于相同的主干架构。与此同时，稀疏表示学习推动传统的倒排索引技术已经看到了越来越多的兴趣，继承了理想的IR先验，如显式词汇匹配。虽然已经提出了一些架构变体，但在这些模型的训练中投入的精力较少。在这项工作中，我们建立在SPLADE -一个基于稀疏扩展的检索器-并通过研究蒸馏，硬否定挖掘以及预训练的语言模型初始化的效果，显示它能够在多大程度上受益于与密集模型相同的训练改进。我们进一步研究了有效性和效率之间的联系，在域和zero-shot设置，导致国家的最先进的结果，在这两种情况下充分表达的模型。
英文摘要：
Neural retrievers based on dense representations combined with Approximate Nearest Neighbors search have recently received a lot of attention, owing their success to distillation and/or better sampling of examples for training -- while still relying on the same backbone architecture. In the meantime, sparse representation learning fueled by traditional inverted indexing techniques has seen a growing interest, inheriting from desirable IR priors such as explicit lexical matching. While some architectural variants have been proposed, a lesser effort has been put in the training of such models. In this work, we build on SPLADE -- a sparse expansion-based retriever -- and show to which extent it is able to benefit from the same training improvements as dense models, by studying the effect of distillation, hard-negative mining as well as the Pre-trained Language Model initialization. We furthermore study the link between effectiveness and efficiency, on in-domain and zero-shot settings, leading to state-of-the-art results in both scenarios for sufficiently expressive models.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
