arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2601.12974cs.CL
收藏

通过评估LLMs在动态牙科临床场景中弥合知识-行动鸿沟

Bridging the Knowledge-Action Gap by Evaluating LLMs in Dynamic Dental Clinical Scenarios

The Second Affiliated Hospital of Harbin Medical University(哈尔滨医科大学附属第二医院);Peking University School and Hospital of Stomatology(北京大学口腔医学院(口腔医院));Medlinker Intelligent and Digital Technology Co., Ltd(医联智数科技有限公司);The University of Queensland(昆士兰大学);Beijing Xicheng District Health Care Hospital for Mothers and Children(北京市西城区妇幼保健院)

Hongyang Ma, Tiantian Gu, Huaiyuan Sun, Huilin Zhu, Yongxin Wang, Jie Li, Wubin Sun, Zeliang Lian, Yinghong Zhou, Yi Gao, Shirui Wang, Zhihui Tang

AI总结:

本文通过SCMPE基准评估LLMs在动态牙科临床场景中的表现,发现其在动态对话中存在主动信息收集和状态跟踪的瓶颈,揭示了外部知识不足以弥补推理差距,需领域自适应预训练。

摘要:

大语言模型(LLMs)从被动的知识检索者向自主临床代理的转变,要求评估方式从静态准确性转向动态行为可靠性。为了探索这一边界,在牙科领域,一个需要高质量AI建议来独特赋能患者参与决策制定的领域,我们提出了标准化临床管理与性能评估(SCMPE)基准,全面评估从知识导向评估(静态客观任务)到基于工作流的模拟(多轮模拟患者互动)。我们的分析发现,尽管模型在静态客观任务中表现出高专业性,但其在动态临床对话中的表现却急剧下降,识别出主要瓶颈不在于知识保留,而在于主动信息收集和动态状态跟踪的关键挑战。映射“指南遵循”与“决策质量”揭示了一般模型普遍存在“高效率、低安全”的风险。此外,我们量化了检索增强生成(RAG)的影响。虽然RAG在静态任务中缓解了幻觉,但在动态工作流中的效果有限且不均一,有时甚至导致退化。这强调了外部知识单独无法弥补推理差距,而需要领域自适应预训练。本研究实证地绘制了牙科LLMs的能力边界,为弥合标准化知识与安全、自主临床实践之间的鸿沟提供了路线图。

英文摘要:

The transition of Large Language Models (LLMs) from passive knowledge retrievers to autonomous clinical agents demands a shift in evaluation-from static accuracy to dynamic behavioral reliability. To explore this boundary in dentistry, a domain where high-quality AI advice uniquely empowers patient-participatory decision-making, we present the Standardized Clinical Management & Performance Evaluation (SCMPE) benchmark, which comprehensively assesses performance from knowledge-oriented evaluations (static objective tasks) to workflow-based simulations (multi-turn simulated patient interactions). Our analysis reveals that while models demonstrate high proficiency in static objective tasks, their performance precipitates in dynamic clinical dialogues, identifying that the primary bottleneck lies not in knowledge retention, but in the critical challenges of active information gathering and dynamic state tracking. Mapping "Guideline Adherence" versus "Decision Quality" reveals a prevalent "High Efficacy, Low Safety" risk in general models. Furthermore, we quantify the impact of Retrieval-Augmented Generation (RAG). While RAG mitigates hallucinations in static tasks, its efficacy in dynamic workflows is limited and heterogeneous, sometimes causing degradation. This underscores that external knowledge alone cannot bridge the reasoning gap without domain-adaptive pre-training. This study empirically charts the capability boundaries of dental LLMs, providing a roadmap for bridging the gap between standardized knowledge and safe, autonomous clinical practice.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑