arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2305.07614cs.IRcs.CL
收藏

NevIR:神经信息检索中的否定

NevIR: Negation in Neural Information Retrieval

Johns Hopkins University(约翰斯·霍普金斯大学)

Orion Weller, Dawn Lawrie, Benjamin Van Durme

AI总结:

本文构建NevIR基准,评估否定对神经信息检索模型排序的影响,发现现有模型大多忽略否定,表现接近随机,并指出继续微调和增大模型规模虽能提升性能,但与人类水平仍有很大差距。

摘要:

否定是一种常见的日常现象,并且一直是语言模型(LMs)的一个持续弱点。尽管信息检索(IR)社区已将语言模型作为现代信息检索架构的骨干,但关于否定如何影响神经信息检索的研究却很少甚至没有。因此,我们围绕这一主题构建了一个直接的基准:要求信息检索模型对仅因否定而不同的两篇文档进行排序。我们表明,结果根据信息检索架构的类型差异很大:交叉编码器表现最好,其次是晚交互模型,排在最后的是双编码器和稀疏神经架构。我们发现,大多数信息检索模型(包括最先进的模型)没有考虑否定,其表现与随机排序相同或更差。我们表明,尽管在一个包含否定的对比文档数据集上继续进行微调这一显而易见的做法能够提高性能(模型规模增大也能提高性能),但机器与人类表现之间仍然存在巨大差距。

英文摘要:

Negation is a common everyday phenomena and has been a consistent area of weakness for language models (LMs). Although the Information Retrieval (IR) community has adopted LMs as the backbone of modern IR architectures, there has been little to no research in understanding how negation impacts neural IR. We therefore construct a straightforward benchmark on this theme: asking IR models to rank two documents that differ only by negation. We show that the results vary widely according to the type of IR architecture: cross-encoders perform best, followed by late-interaction models, and in last place are bi-encoder and sparse neural architectures. We find that most information retrieval models (including SOTA ones) do not consider negation, performing the same or worse than a random ranking. We show that although the obvious approach of continued fine-tuning on a dataset of contrastive documents containing negations increases performance (as does model size), there is still a large gap between machine and human performance.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑