ASTRID——一种用于评估基于RAG的临床问答系统的自动化可扩展三元指标
ASTRID -- An Automated and Scalable TRIaD for the Evaluation of RAG-based Clinical Question Answering Systems
AI总结：
本文提出ASTRID三元指标（CR、RA、CF）用于自动化评估基于RAG的临床问答系统，其中新指标CF能更好衡量对话场景下的忠实性，并通过200多个真实患者问题及九种LLMs验证其与人类评估的一致性。
摘要：
大型语言模型（LLMs）在临床问答（QA）中展现出令人瞩目的潜力，其中检索增强生成（RAG）已成为确保模型响应事实准确性的主要方法。然而，当前的自动化RAG指标在临床和对话式使用场景中表现不佳。使用临床人工评估响应成本高昂、难以扩展，并且不利于RAG系统的持续迭代开发。为应对这些挑战，我们提出了ASTRID——一种用于评估利用RAG的临床QA系统的自动化可扩展三元指标——由三个指标组成：上下文相关性（CR）、拒绝准确性（RA）和对话忠实性（CF）。我们提出的新评估指标CF旨在更好地捕捉模型响应对知识库的忠实程度，同时不会因对话元素而受到惩罚。为验证我们的三元指标，我们构建了一个包含200多个真实世界患者问题的数据集，这些问题是在白内障手术——全球手术量最高的手术——术后随访期间向基于LLM的QA智能体提出的，并补充了临床医生为紧急、临床和非临床域外场景选择的问题。我们证明，在对话式使用场景中，CF比现有定义更能预测人类对忠实性的评分。此外，我们表明，使用由CF、RA和CR组成的三元指标进行评估，与临床医生对不适当、有害或无帮助响应的评估保持一致。最后，我们使用九种不同的LLMs证明，这三个指标能够与人类评估高度一致，凸显了这些指标在LLM驱动的自动化评估流水线中的应用潜力。我们还公开了这些实验的提示和数据集，为进一步的研究和开发提供了宝贵资源。
英文摘要：
Large Language Models (LLMs) have shown impressive potential in clinical question answering (QA), with Retrieval Augmented Generation (RAG) emerging as a leading approach for ensuring the factual accuracy of model responses. However, current automated RAG metrics perform poorly in clinical and conversational use cases. Using clinical human evaluations of responses is expensive, unscalable, and not conducive to the continuous iterative development of RAG systems. To address these challenges, we introduce ASTRID - an Automated and Scalable TRIaD for evaluating clinical QA systems leveraging RAG - consisting of three metrics: Context Relevance (CR), Refusal Accuracy (RA), and Conversational Faithfulness (CF). Our novel evaluation metric, CF, is designed to better capture the faithfulness of a model's response to the knowledge base without penalising conversational elements. To validate our triad, we curate a dataset of over 200 real-world patient questions posed to an LLM-based QA agent during surgical follow-up for cataract surgery - the highest volume operation in the world - augmented with clinician-selected questions for emergency, clinical, and non-clinical out-of-domain scenarios. We demonstrate that CF can predict human ratings of faithfulness better than existing definitions for conversational use cases. Furthermore, we show that evaluation using our triad consisting of CF, RA, and CR exhibits alignment with clinician assessment for inappropriate, harmful, or unhelpful responses. Finally, using nine different LLMs, we demonstrate that the three metrics can closely agree with human evaluations, highlighting the potential of these metrics for use in LLM-driven automated evaluation pipelines. We also publish the prompts and datasets for these experiments, providing valuable resources for further research and development.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
