arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2201.01745cs.IRcs.CL
收藏

原子化搜索长度:超越用户模型

Atomized Search Length: Beyond User Models

Google(谷歌)

John Alex, Keith Hall, Donald Metzler

摘要:

我们认为,目前的IR指标,优化用户体验,测量IR空间的一部分太窄。如果IR系统很弱,这些指标会欠采样或完全过滤掉需要改进的更深层次的文档。如果IR系统相对较强,则这些指标会对更深层次的相关文档进行欠采样,这些文档可能会支持更强大的IR系统,这些IR系统可以在用户可消化的层次结构或文本摘要中呈现数十或数百个相关文档的内容。我们重新分析了过去28年中的70多个TREC跟踪,结果显示大约有一半的顶级文档样本不足,几乎所有的尾部文档样本不足。我们发现,在2020年的深度学习轨道中,神经系统在排名靠前的文档中实际上接近最优,而在尾部文档中仅比BM 25有适度的增益。我们的分析是基于一个简单的新的面向系统的度量,“原子化的搜索长度”,它能够准确,均匀地测量任何深度的所有相关文档。

英文摘要:

We argue that current IR metrics, modeled on optimizing user experience, measure too narrow a portion of the IR space. If IR systems are weak, these metrics undersample or completely filter out the deeper documents that need improvement. If IR systems are relatively strong, these metrics undersample deeper relevant documents that could underpin even stronger IR systems, ones that could present content from tens or hundreds of relevant documents in a user-digestible hierarchy or text summary. We reanalyze over 70 TREC tracks from the past 28 years, showing that roughly half undersample top ranked documents and nearly all undersample tail documents. We show that in the 2020 Deep Learning tracks, neural systems were actually near-optimal at top-ranked documents, compared to only modest gains over BM25 on tail documents. Our analysis is based on a simple new systems-oriented metric, 'atomized search length', which is capable of accurately and evenly measuring all relevant documents at any depth.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑