arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2407.02464cs.IRstat.ML
收藏

使用生成式人工智能进行信息检索评估的可靠置信区间

Reliable Confidence Intervals for Information Retrieval Evaluation Using Generative A.I

Harrie Oosterhuis, Rolf Jagerman, Zhen Qin, Xuanhui Wang, Michael Bendersky

AI总结:

针对生成式 LLM 相关性标注存在系统性错误导致 IR 评估不可靠的问题,提出基于预测驱动推断和保形风险控制的置信区间方法,并证明其优于自助法估计。

摘要:

信息检索(IR)系统的传统评估通常成本高昂,因为它需要人类专家进行人工相关性标注。生成式人工智能——尤其是大语言模型(LLM)——的最新进展能够以相对较小的计算成本大规模生成相关性标注。这有可能减轻传统上与 IR 评估相关的成本,并使其适用于众多低资源应用。然而,生成的相关性标注并非不会出现(系统性)错误,因此,直接将其用于评估会产生不可靠的结果。在本研究中,我们提出了两种基于预测驱动推断和保形风险控制的方法,这些方法利用计算机生成的相关性标注,为 IR 评估指标构建可靠的置信区间(CI)。我们提出的方法需要少量可靠标注,以便这些方法能够从统计上分析生成标注中的错误。利用这些信息,我们可以在具有强理论保证的情况下围绕评估指标构建 CI。与现有方法不同,我们的保形风险控制方法专门为排序指标设计,并且可以按查询和文档改变其 CI。实验结果表明,我们的 CI 能够准确捕捉基于 LLM 标注的评估中的方差和偏差,优于典型的经验自助法估计。我们希望我们的贡献能为许多传统上无法实现可靠评估的 IR 应用带来可靠评估。

英文摘要:

The traditional evaluation of information retrieval (IR) systems is generally very costly as it requires manual relevance annotation from human experts. Recent advancements in generative artificial intelligence -- specifically large language models (LLMs) -- can generate relevance annotations at an enormous scale with relatively small computational costs. Potentially, this could alleviate the costs traditionally associated with IR evaluation and make it applicable to numerous low-resource applications. However, generated relevance annotations are not immune to (systematic) errors, and as a result, directly using them for evaluation produces unreliable results. In this work, we propose two methods based on prediction-powered inference and conformal risk control that utilize computer-generated relevance annotations to place reliable confidence intervals (CIs) around IR evaluation metrics. Our proposed methods require a small number of reliable annotations from which the methods can statistically analyze the errors in the generated annotations. Using this information, we can place CIs around evaluation metrics with strong theoretical guarantees. Unlike existing approaches, our conformal risk control method is specifically designed for ranking metrics and can vary its CIs per query and document. Our experimental results show that our CIs accurately capture both the variance and bias in evaluation based on LLM annotations, better than the typical empirical bootstrapping estimates. We hope our contributions bring reliable evaluation to the many IR applications where this was traditionally infeasible.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑