arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.14151cs.IRcs.AIcs.CLcs.LG
收藏

BIRCO:一个具有复杂目标的信息检索任务基准

BIRCO: A Benchmark of Information Retrieval Tasks with Complex Objectives

Laboratory for Emerging Intelligence(新兴智能实验室);University of California, San Diego(加州大学圣迭戈分校)

Xiaoyue Wang, Jianyou Wang, Weili Cao, Kaicheng Wang, Ramamohan Paturi, Leon Bergen

AI总结:

本文提出BIRCO基准,用于评估IR系统面对多方面复杂用户目标时的文档检索能力,并通过模块化框架发现简单基线可匹敌或超越复杂方法。

摘要:

我们提出了具有复杂目标的信息检索(IR)任务基准(Benchmark of Information Retrieval tasks with Complex Objectives,BIRCO)。BIRCO评估IR系统在给定多方面用户目标时检索文档的能力。该基准的复杂性和紧凑规模使其适合评估基于大型语言模型(LLM)的信息检索系统。我们提出了一个模块化框架,用于研究可能影响LLM在检索任务上表现的因素,并识别出一个简单的基线模型,该模型能够匹敌或超越现有方法以及更复杂的替代方案。没有任何方法能够在所有基准任务上取得令人满意的表现,这表明需要更强的模型和新的检索协议来满足复杂的用户需求。

英文摘要:

We present the Benchmark of Information Retrieval (IR) tasks with Complex Objectives (BIRCO). BIRCO evaluates the ability of IR systems to retrieve documents given multi-faceted user objectives. The benchmark's complexity and compact size make it suitable for evaluating large language model (LLM)-based information retrieval systems. We present a modular framework for investigating factors that may influence LLM performance on retrieval tasks, and identify a simple baseline model which matches or outperforms existing approaches and more complex alternatives. No approach achieves satisfactory performance on all benchmark tasks, suggesting that stronger models and new retrieval protocols are necessary to address complex user needs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑