arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2403.16446cs.CL
收藏

迈向 LLMs 临床能力的自动评估:指标、数据与算法

Towards Automatic Evaluation for LLMs' Clinical Capabilities: Metric, Data, and Algorithm

Ant Group(蚂蚁集团);Shanghai Jiao Tong University School of Medicine Affiliated Renji Hospital(上海交通大学医学院附属仁济医院);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))

Lei Liu, Xiaoyan Yang, Fangzhou Li, Chenfei Chi, Yue Shen, Shiwei Lyu Ming Zhang, Xiaowei Ma, Xiangguo Lyu, Liya Ma, Zhiqiang Zhang, Wei Xue, Yiran Huang, Jinjie Gu

AI总结:

提出一种包含指标、数据和算法的自动评估范式,通过制定LLM特定临床路径、引入标准化病人数据集与多智能体检索增强评估框架,在泌尿外科领域构建基准,有效评估LLMs的临床服务能力。

摘要:

大型语言模型(LLMs)凭借在自然语言建模上的卓越表现,正日益受到关注以提升医疗诊断的临床效率。为确保临床应用的安全可靠,对 LLMs 的评估对于更好地缓解幻觉等潜在风险至关重要。然而,当前评估方法严重依赖费时费力的人工参与以获得符合人类偏好的判断。为克服此挑战,我们提出一种自动评估范式,专为评估 LLMs 提供疾病诊断和治疗等临床服务的能力而设计。该评估范式包含三个基本要素:指标、数据和算法。具体而言,受专业临床实践路径启发,我们制定了 LLM 特定临床路径(LCP),以定义医生智能体应具备的临床能力。随后,引入医学教育中的标准化病人(SPs)作为收集评估医疗数据的指南,这能很好地确保评估流程的完整性。基于这些步骤,我们开发了一个多智能体框架,以模拟 SPs 与医生智能体间的交互环境,该框架配备检索增强评估(RAE),用于判定医生智能体的行为是否符合 LCP。上述范式可扩展至任何类似临床场景,以自动评估 LLMs 的医疗能力。应用该范式,我们在泌尿外科领域构建了评估基准,包括 LCP、SPs 数据集和自动化 RAE。我们开展了大量实验以证明所提方法的有效性,为 LLMs 在临床实践中的安全可靠部署提供更多见解。

英文摘要:

Large language models (LLMs) are gaining increasing interests to improve clinical efficiency for medical diagnosis, owing to their unprecedented performance in modelling natural language. Ensuring the safe and reliable clinical applications, the evaluation of LLMs indeed becomes critical for better mitigating the potential risks, e.g., hallucinations. However, current evaluation methods heavily rely on labor-intensive human participation to achieve human-preferred judgements. To overcome this challenge, we propose an automatic evaluation paradigm tailored to assess the LLMs' capabilities in delivering clinical services, e.g., disease diagnosis and treatment. The evaluation paradigm contains three basic elements: metric, data, and algorithm. Specifically, inspired by professional clinical practice pathways, we formulate a LLM-specific clinical pathway (LCP) to define the clinical capabilities that a doctor agent should possess. Then, Standardized Patients (SPs) from the medical education are introduced as the guideline for collecting medical data for evaluation, which can well ensure the completeness of the evaluation procedure. Leveraging these steps, we develop a multi-agent framework to simulate the interactive environment between SPs and a doctor agent, which is equipped with a Retrieval-Augmented Evaluation (RAE) to determine whether the behaviors of a doctor agent are in accordance with LCP. The above paradigm can be extended to any similar clinical scenarios to automatically evaluate the LLMs' medical capabilities. Applying such paradigm, we construct an evaluation benchmark in the field of urology, including a LCP, a SPs dataset, and an automated RAE. Extensive experiments are conducted to demonstrate the effectiveness of the proposed approach, providing more insights for LLMs' safe and reliable deployments in clinical practice.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑