评估专用LLM对临床决策支持中医生体验的影响:Ask Avo与ChatGPT-4的比较
Evaluating the Impact of a Specialized LLM on Physician Experience in Clinical Decision Support: A Comparison of Ask Avo and ChatGPT-4
AI总结:
本研究在模拟临床场景中比较了专用LLM软件Ask Avo与ChatGPT-4,发现Ask Avo在可信度、可操作性等五项体验指标上均显著优于后者,证明专用LLM能更好提升临床决策支持体验。
摘要:
使用大语言模型(LLMs)来增强临床决策支持系统是一个备受关注且快速发展的课题,但目前存在的缺陷(如幻觉和缺乏清晰的来源引用)使其在临床环境中的应用不可靠。本研究在模拟临床场景环境中,针对医生终端用户体验,评估了Ask Avo与ChatGPT-4的表现。Ask Avo是由AvoMD开发的一款基于LLM的软件,它整合了专有的语言模型增强检索(LMAR)系统、内置视觉引用提示以及专为医生交互设计的提示工程。62名研究参与者向两个模型提出了8个源自不同专科医学指南文件的临床问题,每个回答在可信度、可操作性、相关性、全面性和友好格式方面按1至5分进行评分。Ask Avo在所有标准上均显著优于ChatGPT-4:可信度(4.52 vs. 3.34, p<0.001)、可操作性(4.41 vs. 3.19, p<0.001)、相关性(4.55 vs. 3.49, p<0.001)、全面性(4.50 vs. 3.37, p<0.001)和友好格式(4.52 vs. 3.60, p<0.001)。研究结果表明,专为临床医生需求设计的专用LLM在用户体验上比通用LLM有显著提升。Ask Avo这种基于证据且贴合临床医生需求的方法,在采用LLM增强的临床决策支持软件方面展现出良好前景。
英文摘要:
The use of Large language models (LLMs) to augment clinical decision support systems is a topic with rapidly growing interest, but current shortcomings such as hallucinations and lack of clear source citations make them unreliable for use in the clinical environment. This study evaluates Ask Avo, an LLM-derived software by AvoMD that incorporates a proprietary Language Model Augmented Retrieval (LMAR) system, in-built visual citation cues, and prompt engineering designed for interactions with physicians, against ChatGPT-4 in end-user experience for physicians in a simulated clinical scenario environment. Eight clinical questions derived from medical guideline documents in various specialties were prompted to both models by 62 study participants, with each response rated on trustworthiness, actionability, relevancy, comprehensiveness, and friendly format from 1 to 5. Ask Avo significantly outperformed ChatGPT-4 in all criteria: trustworthiness (4.52 vs. 3.34, p<0.001), actionability (4.41 vs. 3.19, p<0.001), relevancy (4.55 vs. 3.49, p<0.001), comprehensiveness (4.50 vs. 3.37, p<0.001), and friendly format (4.52 vs. 3.60, p<0.001). Our findings suggest that specialized LLMs designed with the needs of clinicians in mind can offer substantial improvements in user experience over general-purpose LLMs. Ask Avo's evidence-based approach tailored to clinician needs shows promise in the adoption of LLM-augmented clinical decision support software.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。