arXiv 2407.158622024-07-16cs.LGcs.AIcs.CLcs.CY

轻量级开源大语言模型在儿科咨询中的性能评估：一项对比分析

Performance Evaluation of Lightweight Open-source Large Language Models in Pediatric Consultations: A Comparative Analysis

Children’s Hospital of Chongqing Medical University（重庆医科大学附属儿童医院）；Stanford University School of Medicine（斯坦福大学医学院）；Chongqing University of Technology（重庆理工大学）；Columbia University（哥伦比亚大学）

Qiuhong Wei, Ying Cui, Mengwei Ding, Yanqin Wang, Lingling Xiang, Zhengxiong Yao, Ceran Chen, Ying Long, Zhezhen Jin, Ximing Xu

备注：: 27 pages in total with 17 pages of main manuscript and 10 pages of supplementary materials; 4 figures in the main manuscript and 2 figures in supplementary material

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本研究评估了轻量级开源大语言模型在儿科咨询中的表现，发现ChatGLM3-6B优于Vicuna系列但仍逊于ChatGPT-3.5，表明轻量级模型有应用潜力但需持续改进。

摘要：

大语言模型（LLMs）在医学领域展现出潜在应用，但数据隐私和计算负担限制了其在医疗机构中的部署。开源和轻量级版本的LLMs作为潜在解决方案出现，但其性能，特别是在儿科环境中的表现仍未被充分探索。在这项横断面研究中，从公共在线医疗论坛随机抽取了250个患者咨询问题，涵盖25个儿科部门，每个部门10个问题，时间跨度为2022年12月1日至2023年10月30日。两款轻量级开源LLMs（ChatGLM3-6B和Vicuna-7B），以及一款较大规模模型Vicuna-13B和广泛使用的专有模型ChatGPT-3.5，于2023年11月1日至7日期间用中文独立回答了这些问题。为评估可重复性，每个咨询均重复一次。我们发现，ChatGLM3-6B在准确性和完整性方面优于Vicuna-13B和Vicuna-7B（P < .001），但均逊于ChatGPT-3.5。在准确性最高评分占比上，ChatGPT-3.5为65.2%，ChatGLM3-6B为41.2%，Vicuna-13B为11.2%，Vicuna-7B为4.4%。在完整性最高评分占比上，ChatGPT-3.5领先（78.4%），其次为ChatGLM3-6B（76.0%）、Vicuna-13B（34.8%）和Vicuna-7B（22.0%）。在可读性上，ChatGLM3-6B与ChatGPT-3.5相当，且均优于Vicuna模型（P < .001）。在同理心方面，ChatGPT-3.5优于轻量级LLMs（P < .001）。在安全性上，所有模型表现相当（P > .05），超过98.4%的回答被评为安全。重复咨询验证了这些结果。结论：轻量级LLMs在儿科医疗保健中展现出应用前景，但与大规模专有LLMs之间的差距凸显了持续开发的必要性。

英文摘要：

Large language models (LLMs) have demonstrated potential applications in medicine, yet data privacy and computational burden limit their deployment in healthcare institutions. Open-source and lightweight versions of LLMs emerge as potential solutions, but their performance, particularly in pediatric settings remains underexplored. In this cross-sectional study, 250 patient consultation questions were randomly selected from a public online medical forum, with 10 questions from each of 25 pediatric departments, spanning from December 1, 2022, to October 30, 2023. Two lightweight open-source LLMs, ChatGLM3-6B and Vicuna-7B, along with a larger-scale model, Vicuna-13B, and the widely-used proprietary ChatGPT-3.5, independently answered these questions in Chinese between November 1, 2023, and November 7, 2023. To assess reproducibility, each inquiry was replicated once. We found that ChatGLM3-6B demonstrated higher accuracy and completeness than Vicuna-13B and Vicuna-7B (P < .001), but all were outperformed by ChatGPT-3.5. ChatGPT-3.5 received the highest ratings in accuracy (65.2%) compared to ChatGLM3-6B (41.2%), Vicuna-13B (11.2%), and Vicuna-7B (4.4%). Similarly, in completeness, ChatGPT-3.5 led (78.4%), followed by ChatGLM3-6B (76.0%), Vicuna-13B (34.8%), and Vicuna-7B (22.0%) in highest ratings. ChatGLM3-6B matched ChatGPT-3.5 in readability, both outperforming Vicuna models (P < .001). In terms of empathy, ChatGPT-3.5 outperformed the lightweight LLMs (P < .001). In safety, all models performed comparably well (P > .05), with over 98.4% of responses being rated as safe. Repetition of inquiries confirmed these findings. In conclusion, Lightweight LLMs demonstrate promising application in pediatric healthcare. However, the observed gap between lightweight and large-scale proprietary LLMs underscores the need for continued development efforts.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。