arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2205.02870cs.IR
收藏

MS偏移:MS MARCO分布偏移对神经检索的影响

MS-Shift: An Analysis of MS MARCO Distribution Shifts on Neural Retrieval

Simon Lupart, Thibault Formal, Stéphane Clinchant

摘要:

预训练语言模型最近出现在信息检索中,作为新一代神经系统的支柱,在各种任务上优于传统方法。然而,目前还不清楚这种方法在zero-shot条件下推广到什么程度。最近的BEIR基准通过比较与训练条件不同的数据集和任务上的模型,为这个问题提供了部分答案。我们的目标是解决同样的问题,比较模型下更明确的分布偏移。为此,我们在MS MARCO中构建了三个基于查询的分布偏移(查询语义,查询意图,查询长度),用于评估基于BERT的三个主要神经检索器家族:稀疏,密集和后期交互-以及单BERT重新排名器。我们进一步分析了训练和测试查询分布之间的性能下降。特别是,我们实验两个泛化指标:第一个基于训练/测试查询词汇重叠,第二个基于表示的训练双编码器。直觉上,这些指标验证了测试集离训练集越远,性能下降越严重。我们还表明,模型对转变的反应不同-密集方法受到的影响最大。总体而言,我们的研究表明,它是可能的设计更可控的分布偏移作为一种工具,以更好地理解IR模型的泛化。最后,我们发布了MS MARCO查询子集,它为信息检索中的基准zero-shot传输提供了额外的资源。

英文摘要:

Pre-trained Language Models have recently emerged in Information Retrieval as providing the backbone of a new generation of neural systems that outperform traditional methods on a variety of tasks. However, it is still unclear to what extent such approaches generalize in zero-shot conditions. The recent BEIR benchmark provides partial answers to this question by comparing models on datasets and tasks that differ from the training conditions. We aim to address the same question by comparing models under more explicit distribution shifts. To this end, we build three query-based distribution shifts within MS MARCO (query-semantic, query-intent, query-length), which are used to evaluate the three main families of neural retrievers based on BERT: sparse, dense, and late-interaction -- as well as a monoBERT re-ranker. We further analyse the performance drops between the train and test query distributions. In particular, we experiment with two generalization indicators: the first one based on train/test query vocabulary overlap, and the second based on representations of a trained bi-encoder. Intuitively, those indicators verify that the further away the test set is from the train one, the worse the drop in performance. We also show that models respond differently to the shifts -- dense approaches being the most impacted. Overall, our study demonstrates that it is possible to design more controllable distribution shifts as a tool to better understand generalization of IR models. Finally, we release the MS MARCO query subsets, which provide an additional resource to benchmark zero-shot transfer in Information Retrieval.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑