arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.07790cs.IR
收藏

在BEIR的Touché 2020论辩检索子集上对神经检索模型的系统性评估

Systematic Evaluation of Neural Retrieval Models on the Touché 2020 Argument Retrieval Subset of BEIR

Nandan Thakur, Luiz Bonifacio, Maik Fröbe, Alexander Bondarenko, Ehsan Kamalloo, Martin Potthast, Matthias Hagen, Jimmy Lin

AI总结:

该研究针对神经检索模型在BEIR基准的Touché 2020论辩检索子集上效果远差于BM25的问题,通过黑盒评估和数据去噪实验分析局限,发现模型短段落偏差与标注缺失问题,去噪后模型效果提升但仍不及BM25,同时公开了代码与增强数据集。

摘要:

神经检索模型的零样本效果通常在BEIR基准(一个由多种信息检索评估数据集组成的集合)上进行评估。有趣的是,此前的研究发现,尤其是在BEIR的子集Touché 2020这一论辩检索任务上,神经检索模型的效果远逊于BM25。但迄今为止,尚未有进一步研究探究论辩检索为何如此“特殊”。为更深入分析神经检索模型各自的潜在局限,我们在Touché 2020数据上开展了一项可复现研究。在研究中,我们聚焦两项实验:(i)黑盒评估(即不重新训练模型),并结合使用检索公理进行理论探索;(ii)涉及事后相关性判断的数据去噪评估。我们的黑盒评估揭示了神经模型在Touché 2020数据上存在检索短段落的固有偏差,同时我们还发现神经模型的不少结果在Touché 2020数据中未被标注相关性。由于许多Touché短段落不具备论辩性,因此本身就是不相关的,且缺失的标注让公平比较变得复杂,我们对Touché 2020数据进行了去噪处理:排除极短段落(少于20个词),并遵循Touché指南用事后标注补充未标注数据。在去噪后的数据上,神经模型的效果在nDCG@10指标上最高提升0.52,但BM25的效果仍然更优。我们的代码和增强后的Touché 2020数据集可在https://github.com/castorini/touche-error-analysis获取。

英文摘要:

The zero-shot effectiveness of neural retrieval models is often evaluated on the BEIR benchmark -- a combination of different IR evaluation datasets. Interestingly, previous studies found that particularly on the BEIR subset Touché 2020, an argument retrieval task, neural retrieval models are considerably less effective than BM25. Still, so far, no further investigation has been conducted on what makes argument retrieval so "special". To more deeply analyze the respective potential limits of neural retrieval models, we run a reproducibility study on the Touché 2020 data. In our study, we focus on two experiments: (i) a black-box evaluation (i.e., no model retraining), incorporating a theoretical exploration using retrieval axioms, and (ii) a data denoising evaluation involving post-hoc relevance judgments. Our black-box evaluation reveals an inherent bias of neural models towards retrieving short passages from the Touché 2020 data, and we also find that quite a few of the neural models' results are unjudged in the Touché 2020 data. As many of the short Touché passages are not argumentative and thus non-relevant per se, and as the missing judgments complicate fair comparison, we denoise the Touché 2020 data by excluding very short passages (less than 20 words) and by augmenting the unjudged data with post-hoc judgments following the Touché guidelines. On the denoised data, the effectiveness of the neural models improves by up to 0.52 in nDCG@10, but BM25 is still more effective. Our code and the augmented Touché 2020 dataset are available at \url{https://github.com/castorini/touche-error-analysis}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑