arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.12883cs.CLcs.AIcs.IR
收藏

BRIGHT:一个现实且具有挑战性的推理密集型检索基准

BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval

The University of Hong Kong(香港大学);Princeton University(普林斯顿大学);Stanford University(斯坦福大学);University of Washington(华盛顿大学);Google Cloud AI Research(谷歌云人工智能研究)
Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han-yu Wang, Haisu Liu, Quan Shi, Zachary S. Siegel, Michael Tang, Ruoxi Sun, Jinsung Yoon… 展开作者

Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han-yu Wang, Haisu Liu, Quan Shi, Zachary S. Siegel, Michael Tang, Ruoxi Sun, Jinsung Yoon, Sercan O. Arik, Danqi Chen, Tao Yu

AI总结:

BRIGHT是一个包含1,384个现实世界查询的推理密集型检索基准,现有最先进检索模型在此表现不佳,而显式推理可显著提升检索性能。

摘要:

现有的检索基准主要由信息寻求型查询(例如,来自搜索引擎的聚合问题)组成,其中基于关键词或语义的检索通常就足够了。然而,许多复杂的现实世界查询需要深入推理来识别相关文档,这超越了表面形式匹配。例如,为编程问题查找文档需要理解所涉及函数的逻辑和语法。为了更好地在如此具有挑战性的查询上对检索进行基准测试,我们引入了BRIGHT,这是第一个需要密集推理来检索相关文档的文本检索基准。我们的数据集包含1,384个跨越不同领域的现实世界查询,如经济学、心理学、数学和编程。这些查询来自自然发生且精心策划的人类数据。广泛的评估表明,即使是最先进的检索模型在BRIGHT上也表现不佳。MTEB排行榜上的领先模型(Muennighoff等人,2023年)SFR-Embedding-Mistral(Meng等人,2024年),在MTEB上达到59.0的nDCG@10分数,在BRIGHT上仅产生18.3的nDCG@10分数。我们表明,在查询中融入显式推理可将检索性能提高多达12.2个百分点。此外,融入来自表现最佳检索器的检索文档可提升问答性能。我们相信,BRIGHT为在更现实和更具挑战性的环境中对检索系统进行未来研究铺平了道路。

英文摘要:

Existing retrieval benchmarks primarily consist of information-seeking queries (e.g., aggregated questions from search engines) where keyword or semantic-based retrieval is usually sufficient. However, many complex real-world queries require in-depth reasoning to identify relevant documents that go beyond surface form matching. For example, finding documentation for a coding question requires understanding the logic and syntax of the functions involved. To better benchmark retrieval on such challenging queries, we introduce BRIGHT, the first text retrieval benchmark that requires intensive reasoning to retrieve relevant documents. Our dataset consists of 1,384 real-world queries spanning diverse domains, such as economics, psychology, mathematics, and coding. These queries are drawn from naturally occurring and carefully curated human data. Extensive evaluation reveals that even state-of-the-art retrieval models perform poorly on BRIGHT. The leading model on the MTEB leaderboard (Muennighoff et al., 2023) SFR-Embedding-Mistral (Meng et al., 2024), which achieves a score of 59.0 nDCG@10,1 produces a score of nDCG@10 of 18.3 on BRIGHT. We show that incorporating explicit reasoning about the query improves retrieval performance by up to 12.2 points. Moreover, incorporating retrieved documents from the top-performing retriever boosts question-answering performance. We believe that BRIGHT paves the way for future research on retrieval systems in more realistic and challenging settings.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑