arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.14302cs.CLcs.AI
收藏

可靠且多样化地评估LLM医学知识掌握程度

Reliable and diverse evaluation of LLM medical knowledge mastery

Tsinghua University(清华大学)

Yuxuan Zhou, Xien Liu, Chen Ning, Xiao Zhang, Ji Wu

AI总结:

本文提出PretexEval框架,通过谓词等价变换动态生成可靠多样的医学知识测试样本,并基于两个临床知识库评估12个知名LLM,揭示其医学知识掌握仍存在显著不足。

摘要:

掌握医学知识对于医学专用LLM至关重要。然而,尽管存在MedQA等医学基准,但仍缺少一个能够充分利用现有知识库来评估LLM医学知识掌握程度的统一框架。在本研究中,我们提出了一个新颖框架PretexEval,该框架可针对任意给定医学知识库动态生成可靠且多样化的测试样本,以评估LLM。我们注意到,通过模板或LLM直接从知识库生成的测试样本可能引入事实错误,并且缺乏多样性。为解决这些问题,我们在所提出的评估框架中引入了一种新颖模式,该模式利用谓词等价变换,为任意给定医学知识点生成一系列变体。最后,这些生成的谓词变体被转换为文本语言,从而产生一系列可靠且多样化的测试样本,用于评估LLM是否完全掌握给定的医学事实知识点。在此,我们使用所提出的框架,基于两个对临床诊断和治疗至关重要的知识库,系统研究了12个知名LLM对医学事实知识的掌握情况。评估结果表明,尽管当前LLM在一些著名公共基准上取得了相当大的成功,但它们在完全掌握医学知识方面仍表现出明显不足。这些新发现为开发医学专用LLM提供了宝贵见解,强调当前LLM在应用于真实世界医学场景之前,迫切需要加强对医学知识全面而深入的掌握。

英文摘要:

Mastering medical knowledge is crucial for medical-specific LLMs. However, despite the existence of medical benchmarks like MedQA, a unified framework that fully leverages existing knowledge bases to evaluate LLMs' mastery of medical knowledge is still lacking. In the study, we propose a novel framework PretexEval that dynamically generates reliable and diverse test samples to evaluate LLMs for any given medical knowledge base. We notice that test samples produced directly from knowledge bases by templates or LLMs may introduce factual errors and also lack diversity. To address these issues, we introduce a novel schema into our proposed evaluation framework that employs predicate equivalence transformations to produce a series of variants for any given medical knowledge point. Finally, these produced predicate variants are converted into textual language, resulting in a series of reliable and diverse test samples to evaluate whether LLMs fully master the given medical factual knowledge point. Here, we use our proposed framework to systematically investigate the mastery of medical factual knowledge of 12 well-known LLMs, based on two knowledge bases that are crucial for clinical diagnosis and treatment. The evaluation results illustrate that current LLMs still exhibit significant deficiencies in fully mastering medical knowledge, despite achieving considerable success on some famous public benchmarks. These new findings provide valuable insights for developing medical-specific LLMs, highlighting that current LLMs urgently need to strengthen their comprehensive and in-depth mastery of medical knowledge before being applied to real-world medical scenarios.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑