arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2203.10232cs.CLcs.IR
收藏

DuReader_retrieval:Web搜索引擎中段落检索的大规模中文基准测试

DuReader_retrieval: A Large-scale Chinese Benchmark for Passage Retrieval from Web Search Engine

University of Edinburgh(爱丁堡大学);Baidu Inc.(百度公司)

Yifu Qiu, Hongyu Li, Yingqi Qu, Ying Chen, Qiaoqiao She, Jing Liu, Hua Wu, Haifeng Wang

摘要:

本文介绍了一个用于段落检索的大规模中文数据集DuReader_retrieval。DuReader_retrieval包含来自商业搜索引擎的超过9万个查询和超过800万个独特的段落。为了减轻其他数据集的缺点并确保我们的基准测试的质量,我们(1)通过手动注释从多个检索器汇集的结果来减少开发和测试集中的假阴性,以及(2)删除语义上与开发和测试查询相似的训练查询。此外,我们还提供了两个用于跨域评估的域外测试集,以及一组用于跨语言检索评估的人工翻译查询。实验结果表明,DuReader_retrieval是一个具有挑战性的和一些问题仍然没有解决,如显着的短语不匹配和查询和段落之间的句法不匹配。这些实验还表明,密集检索器不能很好地推广跨领域,跨语言检索本质上是具有挑战性的。DuReader_retrieval可在https://github.com/baidu/DuReader/tree/master/DuReader-Retrieval上公开获取。

英文摘要:

In this paper, we present DuReader_retrieval, a large-scale Chinese dataset for passage retrieval. DuReader_retrieval contains more than 90K queries and over 8M unique passages from a commercial search engine. To alleviate the shortcomings of other datasets and ensure the quality of our benchmark, we (1) reduce the false negatives in development and test sets by manually annotating results pooled from multiple retrievers, and (2) remove the training queries that are semantically similar to the development and testing queries. Additionally, we provide two out-of-domain testing sets for cross-domain evaluation, as well as a set of human translated queries for for cross-lingual retrieval evaluation. The experiments demonstrate that DuReader_retrieval is challenging and a number of problems remain unsolved, such as the salient phrase mismatch and the syntactic mismatch between queries and paragraphs. These experiments also show that dense retrievers do not generalize well across domains, and cross-lingual retrieval is essentially challenging. DuReader_retrieval is publicly available at https://github.com/baidu/DuReader/tree/master/DuReader-Retrieval.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑