arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2204.11447cs.IR
收藏

神经检索模型的内插和外推性能评价

Evaluating Interpolation and Extrapolation Performance of Neural Retrieval Models

Jingtao Zhan, Xiaohui Xie, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, Shaoping Ma

摘要:

检索模型不仅要对训练数据进行插值,而且要对不同于训练数据的查询进行良好的外推。虽然神经检索模型在ad-hoc搜索基准上表现出令人印象深刻的性能,但我们仍然对它们在内插和外推方面的表现知之甚少。在本文中,我们证明了分别评估神经检索模型的两种能力的重要性。首先,我们从这两个角度研究现有的ad-hoc搜索基准。我们调查了训练数据和测试数据的分布,发现查询实体、查询意图和相关性标签有相当大的重叠。这一发现意味着对这些测试集的评价偏向于内插,不能准确反映外推能力。其次,我们提出了一种新的评估协议,分别评估现有基准数据集上的内插和外推性能。它基于查询相似性对训练和测试数据进行重新采样,并利用重新采样的数据集进行训练和评估。最后,我们利用所提出的评估协议,全面重新审视了一些广泛采用的神经检索模型。结果表明,当从内插到外推时,模型的表现不同。例如,基于表示的检索模型在内插而不是外推方面几乎与基于交互的检索模型一样好。因此,有必要分别评估内插和外推的性能和建议的reserve方法作为一个简单而有效的评估工具,为未来的IR研究。

英文摘要:

A retrieval model should not only interpolate the training data but also extrapolate well to the queries that are different from the training data. While neural retrieval models have demonstrated impressive performance on ad-hoc search benchmarks, we still know little about how they perform in terms of interpolation and extrapolation. In this paper, we demonstrate the importance of separately evaluating the two capabilities of neural retrieval models. Firstly, we examine existing ad-hoc search benchmarks from the two perspectives. We investigate the distribution of training and test data and find a considerable overlap in query entities, query intent, and relevance labels. This finding implies that the evaluation on these test sets is biased toward interpolation and cannot accurately reflect the extrapolation capacity. Secondly, we propose a novel evaluation protocol to separately evaluate the interpolation and extrapolation performance on existing benchmark datasets. It resamples the training and test data based on query similarity and utilizes the resampled dataset for training and evaluation. Finally, we leverage the proposed evaluation protocol to comprehensively revisit a number of widely-adopted neural retrieval models. Results show models perform differently when moving from interpolation to extrapolation. For example, representation-based retrieval models perform almost as well as interaction-based retrieval models in terms of interpolation but not extrapolation. Therefore, it is necessary to separately evaluate both interpolation and extrapolation performance and the proposed resampling method serves as a simple yet effective evaluation tool for future IR studies.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑