从反馈到检查清单：AI 生成临床笔记的基于实证的评估
From Feedback to Checklists: Grounded Evaluation of AI-Generated Clinical Notes
AI总结：
本文提出从真实用户反馈中提炼结构化检查清单的方法，用于评估 AI 生成的临床笔记，实验证明其在覆盖率、多样性和预测能力上优于基线，并与临床医生偏好高度一致。
摘要：
AI 生成的临床笔记在医疗保健中的应用日益增多，但由于专家评审的高度主观性和可扩展性有限，评估其质量仍是一项挑战。现有的自动化指标往往无法与真实世界中医生的偏好保持一致。为解决这一问题，我们提出了一种流程，系统地将真实用户反馈提炼为用于笔记评估的结构化检查清单。这些检查清单设计为可解释的、基于人类反馈的，并可由基于 LLM 的评估器强制执行。利用来自一个已部署的 AI 医疗抄写系统的超过 21,000 次临床接触的去标识化数据（根据 HIPAA 安全港标准准备），我们表明，在我们的离线评估中，基于反馈的检查清单在覆盖率、多样性和对人工评分的预测能力方面优于基线方法。大量实验证实了检查清单对质量退化扰动的鲁棒性、与临床医生偏好的显著一致性，以及作为评估方法的实用价值。在离线研究环境中，我们的检查清单提供了一种实用工具，用于标记可能未达到我们定义的质量标准的笔记。
英文摘要：
AI-generated clinical notes are increasingly used in healthcare, but evaluating their quality remains a challenge due to high subjectivity and limited scalability of expert review. Existing automated metrics often fail to align with real-world physician preferences. To address this, we propose a pipeline that systematically distills real user feedback into structured checklists for note evaluation. These checklists are designed to be interpretable, grounded in human feedback, and enforceable by LLM-based evaluators. Using deidentified data from over 21,000 clinical encounters (prepared in accordance with the HIPAA safe harbor standard) from a deployed AI medical scribe system, we show that our feedback-derived checklist outperforms a baseline approach in our offline evaluations in coverage, diversity, and predictive power for human ratings. Extensive experiments confirm the checklist's robustness to quality-degrading perturbations, significant alignment with clinician preferences, and practical value as an evaluation methodology. In offline research settings, our checklist offers a practical tool for flagging notes that may fall short of our defined quality standards.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
