arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2605.20591cs.CLcs.CY
收藏

有害吗?网络部署医疗大语言模型中的幻觉与作用层面滥用

Do No Harm? Hallucination and Actor-Level Abuse in Web-Deployed Medical Large Language Models

The University of New South Wales, Sydney, Australia(新南威尔士大学,悉尼,澳大利亚)

Sunday Oyinlola Ogundoyin, Muhammad Ikram, Rahat Masood

AI总结:

本文研究了网络部署的医疗大语言模型中的幻觉和作用层面滥用问题,通过评估6233个MedGPT和10个开源LLM,发现25-30%的MedGPT事实准确性较低,33.6-54.3%违反操作阈值,57.06%的Action-enabled模型缺乏充分的隐私披露,揭示了系统性漏洞,强调了多指标评估和更强的安全保障的必要性。

摘要:

医疗大语言模型(LLMs),包括定制医疗GPT(MedGPTs)和开源模型,正越来越多地部署在网页平台上以提供临床指导。然而,它们存在幻觉、政策不合规和不安全设计的风险。我们对6,233个MedGPT进行了大规模评估,评估了1,500个分层样本以及10个开源LLM。我们引入了两个框架:MedGPT-HEval用于幻觉检测,以及一个基于LLM的流程用于评估违规行为和开发者意图。我们的结果表明,25-30%的MedGPT事实准确性较低,底层和中层模型风险最高;33.6-54.3%违反操作阈值,57.06%的Action-enabled模型缺乏充分的隐私披露。与开源模型相比,MedGPT在事实准确性和语义对齐方面表现更好,但开源模型更稳定。这些结果揭示了幻觉和合规性的系统性缺口,强调了多指标评估和更强的安全保障的必要性。我们发布了HAA-MedGPT,一个结构化数据集,支持未来关于网络面向医疗LLM安全性的研究。

英文摘要:

Medical large language models (LLMs), including custom medical GPTs (MedGPTs) and open-source models, are increasingly deployed on web platforms to provide clinical guidance. However, they pose risks of hallucination, policy noncompliance, and unsafe design. We conduct a large-scale assessment of 6,233 MedGPTs, evaluating a stratified sample of 1,500, together with 10 open-source LLMs. We introduce two frameworks: MedGPT-HEval for hallucination detection and an LLM-based pipeline for assessing policy violations and developer intent. Our results show that 25-30% of MedGPTs exhibit low factual accuracy, with bottom- and middle-tier models at highest risk; 33.6-54.3% violate operational thresholds, and 57.06% of Action-enabled models lack adequate privacy disclosures. Compared with open-source models, MedGPTs achieve higher factual accuracy and semantic alignment, though open-source models are more stable. These results reveal systemic gaps in hallucination and compliance, highlighting the need for multi-metric evaluation and stronger safeguards. We release HAA-MedGPT, a structured dataset that supports future research on the safety of web-facing medical LLMs.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑