当AI出错时:AI辅助用药决策系统中的可靠性与风险
When AI Gets it Wrong: Reliability and Risk in AI-Assisted Medication Decision Systems
AI总结:
本文研究了AI辅助用药系统在现实决策中的可靠性问题,通过模拟药物相互作用和剂量决策场景,分析系统故障类型及其潜在临床影响,强调在安全关键领域如药房实践中,需补充传统性能指标的风险意识评估方法。
摘要:
人工智能(AI)系统日益被整合到医疗和药房工作中,支持药物推荐、剂量确定和药物相互作用检测等任务。尽管这些系统在标准评估指标下通常表现良好,但其在现实决策中的可靠性仍不够理解。在高风险领域如用药管理中,单个错误推荐可能导致严重患者伤害。本文通过聚焦系统故障及其潜在临床后果,探讨AI辅助用药系统的可靠性。不同于仅通过聚合指标评估性能,本文关注错误发生的方式以及AI系统产生错误输出时的情况。通过一系列受控的模拟场景,分析不同类型的系统故障,包括遗漏相互作用、错误风险标记和不适当的剂量推荐。研究发现,AI在用药相关情境中的错误可能导致不良药物反应、无效治疗或延误护理,尤其是在缺乏充分人类监督的情况下。此外,本文讨论了过度依赖AI推荐的风险以及决策过程透明度有限带来的挑战。本文为医疗领域AI评估提供了以可靠性为核心的视角,强调理解故障行为和现实影响的重要性。它突显了在安全关键领域如药房实践中,需补充传统性能指标的风险意识评估方法的必要性。
英文摘要:
Artificial intelligence (AI) systems are increasingly integrated into healthcare and pharmacy workflows, supporting tasks such as medication recommendations, dosage determination, and drug interaction detection. While these systems often demonstrate strong performance under standard evaluation metrics, their reliability in real-world decision-making remains insufficiently understood. In high-risk domains such as medication management, even a single incorrect recommendation can result in severe patient harm. This paper examines the reliability of AI-assisted medication systems by focusing on system failures and their potential clinical consequences. Rather than evaluating performance solely through aggregate metrics, this work shifts attention towards how errors occur and what happens when AI systems produce incorrect outputs. Through a series of controlled, simulated scenarios involving drug interactions and dosage decisions, we analyse different types of system failures, including missed interactions, incorrect risk flagging, and inappropriate dosage recommendations. The findings highlight that AI errors in medication-related contexts can lead to adverse drug reactions, ineffective treatment, or delayed care, particularly when systems are used without sufficient human oversight. Furthermore, the paper discusses the risks of over-reliance on AI recommendations and the challenges posed by limited transparency in decision-making processes. This work contributes a reliability-focused perspective on AI evaluation in healthcare, emphasising the importance of understanding failure behavior and real-world impact. It highlights the need to complement traditional performance metrics with risk-aware evaluation approaches, particularly in safety-critical domains such as pharmacy practice.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。