arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.00309cs.IR
收藏

一种超越传统相关性判断的基于考试的评估方法

An Exam-based Evaluation Approach Beyond Traditional Relevance Judgments

Naghmeh Farzi, Laura Dietz

AI总结:

本研究提出一种不依赖相关性判断的IR评估新范式,通过考试题库与LLM问答系统自动评分,设计两种评估指标,支持题库扩展与未来信息系统的持续评估。

摘要:

当前信息检索(IR)评估以相关性判断为基础,这些判断由人工或自动生成,决策过程则交由大语言模型(LLM)完成。我们提出了一种全新的替代范式,它从不依赖任何形式的相关性判断。相反,若一段文本包含能够解答关键问题的信息,就被定义为相关。我们基于这一思路设计了EXAM可回答性指标,用于评估信息检索/生成系统提供主题相关信息的能力。我们设想由人类评估者编辑并定义一套考试题库,用于测试文本中是否存在相关信息。我们会先生成初始的考试题目集,为这一步骤提供支持。在接下来的阶段,一个基于LLM的问答系统会通过追踪哪些考试题目能被哪些系统响应解答,来自动为系统响应打分。我们提出了两种评估指标:面向召回率的EXAM Cover指标,以及面向精确率的EXAM Qrels指标,后者可通过trec_eval实现。该范式不仅支持事后扩展考试题目集,还便于对未来的信息系统进行持续评估,无论这些系统侧重检索、生成,还是两者兼具。

英文摘要:

Current IR evaluation is based on relevance judgments, created either manually or automatically, with decisions outsourced to Large Language Models (LLMs). We offer an alternative paradigm, that never relies on relevance judgments in any form. Instead, a text is defined as relevant if it contains information that enables the answering of key questions. We use this idea to design the EXAM Answerability Metric to evaluate information retrieval/generation systems for their ability to provide topically relevant information. We envision the role of a human judge to edit and define an exam question bank that will test for the presence of relevant information in text. We support this step by generating an initial set of exam questions. In the next phase, an LLM-based question answering system will automatically grade system responses by tracking which exam questions are answerable with which system responses. We propose two evaluation measures, the recall-oriented EXAM Cover metric, and the precision-oriented EXAM Qrels metric, the latter which can be implemented with trec_eval. This paradigm not only allows for the expansion of the exam question set post-hoc but also facilitates the ongoing evaluation of future information systems, whether they focus on retrieval, generation, or both.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑