arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2301.10035cs.HC
收藏

将ChatGPT的医疗建议进行(图灵)测试

Putting ChatGPT's Medical Advice to the (Turing) Test

Oded Nov, Nina Singh, Devin Mann

摘要:

目的:评估使用ChatGPT或类似的基于AI的聊天机器人进行患者与提供者沟通的可行性。参与者:美国代表性样本,430名18岁及以上的研究参与者。53.2%的受访者是女性;她们的平均年龄为47.1岁。暴露:从EHR中提取了10个代表性的非管理性患者-提供者交互。患者的问题被放置在ChatGPT中,并要求聊天机器人使用与人类提供者的响应大致相同的字数进行响应。在调查中,每个患者的问题后面都有一个提供者或ChatGPT生成的回答。参与者被告知,五个响应是提供者生成的,五个是聊天机器人生成的。参与者被要求,并在经济上激励,以正确地识别响应源。参与者还被问及他们对聊天机器人在患者与提供者沟通中的功能的信任,使用1-5的Likert量表。结果:不同问题的回答正确率在49.0%~ 85.7%之间。平均而言,聊天机器人的响应在65.5%的时间内被正确识别,而提供商的响应在65.1%的时间内被正确区分。平均而言,患者对聊天机器人功能的信任反应呈弱阳性(平均Likert评分:3.4),随着问题中与健康相关的任务复杂性的增加,信任度会降低。结论:ChatGPT对患者问题的回答与提供者的回答难以区分。外行人似乎相信使用聊天机器人来回答风险较低的健康问题。

英文摘要:

Objective: Assess the feasibility of using ChatGPT or a similar AI-based chatbot for patient-provider communication. Participants: A US representative sample of 430 study participants aged 18 and above. 53.2% of respondents analyzed were women; their average age was 47.1. Exposure: Ten representative non-administrative patient-provider interactions were extracted from the EHR. Patients' questions were placed in ChatGPT with a request for the chatbot to respond using approximately the same word count as the human provider's response. In the survey, each patient's question was followed by a provider- or ChatGPT-generated response. Participants were informed that five responses were provider-generated and five were chatbot-generated. Participants were asked, and incentivized financially, to correctly identify the response source. Participants were also asked about their trust in chatbots' functions in patient-provider communication, using a Likert scale of 1-5. Results: The correct classification of responses ranged between 49.0% to 85.7% for different questions. On average, chatbot responses were correctly identified 65.5% of the time, and provider responses were correctly distinguished 65.1% of the time. On average, responses toward patients' trust in chatbots' functions were weakly positive (mean Likert score: 3.4), with lower trust as the health-related complexity of the task in questions increased. Conclusions: ChatGPT responses to patient questions were weakly distinguishable from provider responses. Laypeople appear to trust the use of chatbots to answer lower risk health questions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑