自动化评估可区分优质与劣质的AI对住院患者问题的回应
Automated Evaluation can Distinguish the Good and Bad AI Responses to Patient Questions about Hospitalization
AI总结:
本文通过系统研究,验证了自动化评估在区分AI对患者住院问题回应质量上的有效性,提出了基于临床参考答案的评估方法,展示了自动化评估在提升AI系统比较和患者沟通中的潜力。
摘要:
自动化回答患者提出的健康问题的方法正在兴起,但选择系统需要可靠的评估。目前的黄金标准是人工专家审查,但这种过程劳动密集且缓慢,限制了可扩展性。自动化指标具有潜力,但往往与人类判断不一致且依赖上下文。为了解决自动化评估AI对住院相关问题回应的可行性,我们进行了大规模系统研究。在100个患者案例中,我们收集了28个AI系统的回应(总计2800个),并沿三个维度评估:是否回答问题、是否适当使用临床笔记证据、是否使用一般医学知识。使用临床作者参考答案来锚定指标,自动化排名与人类评分高度一致。我们的发现表明,精心设计的自动化评估可以扩展AI系统的比较评估,并支持患者与医生的沟通。
英文摘要:
Automated approaches to answer patient-posed health questions are rising, but selecting among systems requires reliable evaluation. The current gold standard for evaluating the free-text artificial intelligence (AI) responses--human expert review--is labor-intensive and slow, limiting scalability. Automated metrics are promising yet variably aligned with human judgments and often context-dependent. To address the feasibility of automating the evaluation of AI responses to hospitalization-related questions posed by patients, we conducted a large systematic study of evaluation approaches. Across 100 patient cases, we collected responses from 28 AI systems (2800 total) and assessed them along three dimensions: whether a system response (1) answers the question, (2) appropriately uses clinical note evidence, and (3) uses general medical knowledge. Using clinician-authored reference answers to anchor metrics, automated rankings closely matched human ratings. Our findings suggest that carefully designed automated evaluation can scale comparative assessment of AI systems and support patient-clinician communication.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。