MedKGEval：面向临床大语言模型开放式患者交互的基于知识图谱的多轮评估框架
MedKGEval: A Knowledge Graph-Based Multi-Turn Evaluation Framework for Open-Ended Patient Interactions with Clinical LLMs
AI总结：
MedKGEval提出基于知识图谱的多轮评估框架，通过患者模拟和回合级裁判，识别临床LLM在开放式交互中的行为缺陷与安全风险。
摘要：
在医疗应用中，大语言模型（LLM）的可靠评估仍然是一个悬而未决的挑战，尤其是在捕捉真实临床环境中多轮医患互动的复杂性方面。现有评估方法通常依赖于对完整对话记录的事后审查，从而忽视了医疗对话的动态性、情境敏感性以及患者不断变化的信息需求。在本工作中，我们提出了MedKGEval，一个基于结构化医学知识的新型临床LLM多轮评估框架。我们的方法包含三个关键贡献：（1）一种知识图谱驱动的患者模拟机制，其中专用控制模块从精选知识图谱中检索相关医学事实，从而使患者智能体具备类人的、逼真的对话行为。该知识图谱通过整合开源资源以及从专家标注数据集中提取的额外三元组构建而成；（2）一种原位、回合级评估框架，在对话进行过程中，每个模型响应由裁判智能体根据一组细粒度、任务特定的指标评估其临床适当性、事实正确性和安全性；（3）一个包含八个最先进LLM的全面多轮基准测试，证明了MedKGEval能够识别传统评估流程常忽视的细微行为缺陷和安全风险。尽管最初设计用于中文和英文医疗应用，我们的框架可通过切换输入知识图谱轻松扩展到其他语言，确保无缝的双语支持和领域特定适用性。
英文摘要：
The reliable evaluation of large language models (LLMs) in medical applications remains an open challenge, particularly in capturing the complexity of multi-turn doctor-patient interactions that unfold in real clinical environments. Existing evaluation methods typically rely on post hoc review of full conversation transcripts, thereby neglecting the dynamic, context-sensitive nature of medical dialogues and the evolving informational needs of patients. In this work, we present MedKGEval, a novel multi-turn evaluation framework for clinical LLMs grounded in structured medical knowledge. Our approach introduces three key contributions: (1) a knowledge graph-driven patient simulation mechanism, where a dedicated control module retrieves relevant medical facts from a curated knowledge graph, thereby endowing the patient agent with human-like and realistic conversational behavior. This knowledge graph is constructed by integrating open-source resources with additional triples extracted from expert-annotated datasets; (2) an in-situ, turn-level evaluation framework, where each model response is assessed by a Judge Agent for clinical appropriateness, factual correctness, and safety as the dialogue progresses using a suite of fine-grained, task-specific metrics; (3) a comprehensive multi-turn benchmark of eight state-of-the-art LLMs, demonstrating MedKGEval's ability to identify subtle behavioral flaws and safety risks that are often overlooked by conventional evaluation pipelines. Although initially designed for Chinese and English medical applications, our framework can be readily extended to additional languages by switching the input knowledge graphs, ensuring seamless bilingual support and domain-specific applicability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
