审核患者教育中生成式人工智能的机构异质性：对102本美国移植手册的大规模研究

中文摘要 AI 辅助

医疗系统正在迅速部署生成式人工智能助手，根据机构编写的教育材料回答患者问题，前提是基于本地内容能产生一致的指导。这是否可行取决于一个此前未大规模衡量的问题：基础文档本身是否一致？我们使用结构化输出大语言模型评判器，对来自美国23个实体器官移植中心的102本患者教育手册中的5730465对比较进行审核，并结合1115个患者提出的问题（移植问答）。有四个发现直接关系到部署：（1）机构编辑风格在统计上超越了器官类型界限，同一中心的手册在不同器官间的一致性高于不同中心同一器官的手册（p = 0.0056）；（2）信息差距不成比例地落在代表性不足亚组的核心主题上，生殖健康面临双重风险：它是最常被忽视的主题（82%缺失），而出现时被评判的临床意义最高（86%高显著性分歧）；（3）分歧主题聚集成991个主题，免疫抑制和怀孕时机是风险最高的主题之一；（4）仅从问题框架就能预测每对之间的分歧（AUC = 0.77）。我们讨论了在移植护理中部署面向患者的生成式人工智能的影响。

英文摘要

Health systems are rapidly deploying generative-AI assistants that answer patient questions from institution-authored education materials, on the premise that grounding in local content yields consistent guidance. Do the underlying documents themselves agree? We use a structured-output large-language-model judge to audit 1{,}772{,}261 pairwise comparisons across 102 patient-education handbooks from 23 US solid-organ transplant centers, paired with 1{,}115 patient-derived questions (TransplantQA). Four findings bear directly on deployment: (1) same-center cross-organ agreement exceeds cross-center same-organ agreement by $0.024$ in the primary analysis (Holm-adjusted $p=0.011$), with sensitivity to document selection; (2) information gaps concern topics relevant to underrepresented subgroups, with reproductive health a \emph{double jeopardy}: 82\% absence and 86\% judge-rated high significance among divergent/contradictory pairs; (3) judge-derived themes form 991 clusters, with immunosuppression and pregnancy timing among the highest judge-rated priorities; (4) question and observed-coverage features predict high-divergence questions retrospectively (AUC $0.77$). We discuss implications for deploying patient-facing generative AI in transplant care.

发表机构

• Carnegie Mellon University（卡内基梅隆大学）

机构由 AI 辅助整理，请以论文原文为准。