arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2309.09403cs.IRcs.AI
收藏

为零样本搜索选择使用哪个稠密检索器

Selecting which Dense Retriever to use for Zero-Shot Search

The University of Queensland(昆士兰大学);Neusoft(东软集团)

Ekaterina Khramtsova, Shengyao Zhuang, Mahsa Baktashmotlagh, Xi Wang, Guido Zuccon

AI总结:

本文提出零样本场景下如何选择稠密检索模型的新问题,通过实验证明现有无监督性能评估方法在该场景下失效,并公开了实现代码与分析脚本。

摘要:

我们提出一个新问题:在一个没有标签可用的新集合上进行搜索时,即在零样本设置下,如何选择使用哪个稠密检索模型。许多稠密检索模型都是现成可用的。然而,每个模型的特点是其搜索有效性差异很大——不仅在用于学习稠密表示的数据集的测试部分上如此,更重要的是,在未用于学习稠密表示的不同数据集上也是如此。这是因为稠密检索器通常需要在大量有标签数据上进行训练,才能在特定数据集或领域中达到令人满意的搜索有效性。此外,稠密检索器在训练期间能够观察到标签的数据集上获得的有效性提升,并不一定能泛化到训练期间未观察到的数据集上。然而这是一个困难的问题:通过实证实验,我们表明,受计算机视觉和机器学习领域中存在领域偏移的无监督性能评估近期工作启发的方法,在我们的设置中并不能有效选择高性能的稠密检索器。在零样本设置下,如果存在可靠的稠密检索模型选择方法,且不需要为评估收集标签,将有助于推动稠密检索的广泛采用。因此,我们认为这是信息检索社区应当考虑的一个重要新问题。方法实现、原始结果文件和分析脚本已在 https://www.github.com/anonymized 公开提供。

英文摘要:

We propose the new problem of choosing which dense retrieval model to use when searching on a new collection for which no labels are available, i.e. in a zero-shot setting. Many dense retrieval models are readily available. Each model however is characterized by very differing search effectiveness -- not just on the test portion of the datasets in which the dense representations have been learned but, importantly, also across different datasets for which data was not used to learn the dense representations. This is because dense retrievers typically require training on a large amount of labeled data to achieve satisfactory search effectiveness in a specific dataset or domain. Moreover, effectiveness gains obtained by dense retrievers on datasets for which they are able to observe labels during training, do not necessarily generalise to datasets that have not been observed during training. This is however a hard problem: through empirical experimentation we show that methods inspired by recent work in unsupervised performance evaluation with the presence of domain shift in the area of computer vision and machine learning are not effective for choosing highly performing dense retrievers in our setup. The availability of reliable methods for the selection of dense retrieval models in zero-shot settings that do not require the collection of labels for evaluation would allow to streamline the widespread adoption of dense retrieval. This is therefore an important new problem we believe the information retrieval community should consider. Implementation of methods, along with raw result files and analysis scripts are made publicly available at https://www.github.com/anonymized.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑