RareBench：大语言模型能否充当罕见病专家？
RareBench: Can LLMs Serve as Rare Diseases Specialists?
AI总结：
本文提出RareBench基准和最大罕见病患者开源数据集，系统评估GPT-4等大语言模型在罕见病诊断中的能力，并通过动态少样本提示结合知识图谱显著提升诊断性能，证明LLMs有望辅助罕见病临床诊断。
摘要：
通用大语言模型（LLMs），如GPT-4，已在包括医疗诊断在内的多个领域展现出可观的前景。罕见病影响全球约3亿人，其临床诊断率往往不尽如人意，主要原因是缺乏经验丰富的医生以及区分众多罕见病的复杂性。在此背景下，近期新闻如“ChatGPT在17名医生失败后正确诊断出一名4岁儿童的罕见病”凸显了LLMs在罕见病临床诊断中尚未被充分探索的潜力。为弥合这一研究空白，我们引入了RareBench，这是一个开创性的基准，旨在系统评估LLMs在罕见病领域4个关键维度上的能力。同时，我们编制了最大的罕见病患者开源数据集，为该领域未来研究确立了基准。为促进罕见病的鉴别诊断，我们开发了一种动态少样本提示方法，利用由多个知识库综合而成的全面罕见病知识图谱，显著提升了LLMs的诊断性能。此外，我们对GPT-4与专科医生的诊断能力进行了详尽的比较研究。我们的实验发现强调了将LLMs整合到罕见病临床诊断流程中的巨大潜力，为该领域未来的进展开辟了令人振奋的可能性。
英文摘要：
Generalist Large Language Models (LLMs), such as GPT-4, have shown considerable promise in various domains, including medical diagnosis. Rare diseases, affecting approximately 300 million people worldwide, often have unsatisfactory clinical diagnosis rates primarily due to a lack of experienced physicians and the complexity of differentiating among many rare diseases. In this context, recent news such as "ChatGPT correctly diagnosed a 4-year-old's rare disease after 17 doctors failed" underscore LLMs' potential, yet underexplored, role in clinically diagnosing rare diseases. To bridge this research gap, we introduce RareBench, a pioneering benchmark designed to systematically evaluate the capabilities of LLMs on 4 critical dimensions within the realm of rare diseases. Meanwhile, we have compiled the largest open-source dataset on rare disease patients, establishing a benchmark for future studies in this domain. To facilitate differential diagnosis of rare diseases, we develop a dynamic few-shot prompt methodology, leveraging a comprehensive rare disease knowledge graph synthesized from multiple knowledge bases, significantly enhancing LLMs' diagnostic performance. Moreover, we present an exhaustive comparative study of GPT-4's diagnostic capabilities against those of specialist physicians. Our experimental findings underscore the promising potential of integrating LLMs into the clinical diagnostic process for rare diseases. This paves the way for exciting possibilities in future advancements in this field.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
