arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.03023cs.CLcs.HC
收藏

MedDialogRubrics: 一种用于大型语言模型多轮医疗咨询的综合基准和评估框架

MedDialogRubrics: A Comprehensive Benchmark and Evaluation Framework for Multi-turn Medical Consultations in Large Language Models

Ant Group(蚂蚁集团)

Lecheng Gong, Weimin Fang, Ting Yang, Dongjie Tao, Chunxiao Guo, Peng Wei, Bo Xie, Jinqun Guan, Zixiao Chen, Fang Shi, Jinjie Gu, Junwei Liu

AI总结:

MedDialogRubrics提出了一种用于评估大型语言模型多轮医疗对话能力的综合基准和评估框架,通过合成患者案例和细粒度评估 rubrics,揭示了当前模型在医疗对话管理方面的不足。

摘要:

医疗对话人工智能(AI)在开发更安全有效的医疗对话系统中起着关键作用。然而,现有用于评估医疗大型语言模型(LLMs)信息收集和诊断推理能力的基准和评估框架尚未经过严格评估。为了解决这些差距,我们提出了MedDialogRubrics,一个包含5200个合成患者案例和超过60000个经过LLMs生成并由临床专家进一步完善后的细粒度评估 rubrics 的新基准,专门用于评估LLMs的多轮诊断能力。我们的框架采用多智能体系统来合成真实的患者记录和主诉,而无需访问真实世界电子健康记录,从而缓解隐私和数据治理问题。我们设计了一个受限于一组基本医学事实的稳健患者代理,并辅以动态指导机制,该机制在整个对话过程中持续检测和纠正幻觉,确保模拟案例的内部一致性和临床合理性。此外,我们提出了一种结构化的LLM基于和专家注释的rubric生成管道,该管道检索循证医学(EBM)指南,并利用拒绝采样方法推导出每个案例的优先级rubric项目(

英文摘要:

Medical conversational AI (AI) plays a pivotal role in the development of safer and more effective medical dialogue systems. However, existing benchmarks and evaluation frameworks for assessing the information-gathering and diagnostic reasoning abilities of medical large language models (LLMs) have not been rigorously evaluated. To address these gaps, we present MedDialogRubrics, a novel benchmark comprising 5,200 synthetically constructed patient cases and over 60,000 fine-grained evaluation rubrics generated by LLMs and subsequently refined by clinical experts, specifically designed to assess the multi-turn diagnostic capabilities of LLM. Our framework employs a multi-agent system to synthesize realistic patient records and chief complaints from underlying disease knowledge without accessing real-world electronic health records, thereby mitigating privacy and data-governance concerns. We design a robust Patient Agent that is limited to a set of atomic medical facts and augmented with a dynamic guidance mechanism that continuously detects and corrects hallucinations throughout the dialogue, ensuring internal coherence and clinical plausibility of the simulated cases. Furthermore, we propose a structured LLM-based and expert-annotated rubric-generation pipeline that retrieves Evidence-Based Medicine (EBM) guidelines and utilizes the reject sampling to derive a prioritized set of rubric items ("must-ask" items) for each case. We perform a comprehensive evaluation of state-of-the-art models and demonstrate that, across multiple assessment dimensions, current models face substantial challenges. Our results indicate that improving medical dialogue will require advances in dialogue management architectures, not just incremental tuning of the base-model.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑