arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2011.00696cs.CLcs.IR
收藏

ABNIRML:分析神经信息检索模型的行为

ABNIRML: Analyzing the Behavior of Neural IR Models

University of Glasgow(格拉斯哥大学);Allen Institute for AI(艾伦人工智能研究所);University of Washington(华盛顿大学)

Sean MacAvaney, Sergey Feldman, Nazli Goharian, Doug Downey, Arman Cohan

AI总结:

针对预训练语言模型在即席搜索中效果优异但行为机制不明的问题,本文提出ABNIRML分析框架,通过新型诊断探测测试多项特征,经实证研究厘清模型性能提升因素并识别出潜在偏差。

摘要:

BERT、T5等预训练上下文语言模型已在即席搜索(ad-hoc search)领域创下新的当前最优性能(state-of-the-art)。但目前人们仍未充分理解这些方法效果优异的原因、部分变体效果更优的关键,以及它们可能存在的缺陷。我们提出了一个全新的综合性框架——神经信息检索模型行为分析框架(Analyzing the Behavior of Neural IR ModeLs, ABNIRML),该框架包含新型诊断探测项,可测试写作风格、事实性、对释义与词序的敏感度等多项过往技术未覆盖的特征。为验证该框架的价值,我们开展了大规模实证研究,厘清了助力神经模型性能提升的各类因素,同时识别出模型存在的潜在非预期偏差。部分结果印证了传统认知,例如近年的神经排序模型对查询词精确重叠的依赖度更低,转而利用更丰富的语言信息,这一点可通过其对词序、句序更高的敏感度佐证。另一些结果则更出人意料,比如T5、ColBERT等模型会偏向事实正确(而非仅具备相关性)的文本。此外,即便基于相同的基础语言模型,部分特征也会存在差异,还有些特征可能由模型训练过程中的随机波动产生。

英文摘要:

Pretrained contextualized language models such as BERT and T5 have established a new state-of-the-art for ad-hoc search. However, it is not yet well-understood why these methods are so effective, what makes some variants more effective than others, and what pitfalls they may have. We present a new comprehensive framework for Analyzing the Behavior of Neural IR ModeLs (ABNIRML), which includes new types of diagnostic probes that allow us to test several characteristics -- such as writing styles, factuality, sensitivity to paraphrasing and word order -- that are not addressed by previous techniques. To demonstrate the value of the framework, we conduct an extensive empirical study that yields insights into the factors that contribute to the neural model's gains, and identify potential unintended biases the models exhibit. Some of our results confirm conventional wisdom, like that recent neural ranking models rely less on exact term overlap with the query, and instead leverage richer linguistic information, evidenced by their higher sensitivity to word and sentence order. Other results are more surprising, such as that some models (e.g., T5 and ColBERT) are biased towards factually correct (rather than simply relevant) texts. Further, some characteristics vary even for the same base language model, and other characteristics can appear due to random variations during model training.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑