Trustworthy clinical advice is crucial but can be burdensome to obtain. Limited access and financial costs may lead people to self-diagnose. However, self-diagnosis requires considerable learning and can create risks when people pursue treatment without professional guidance. Large language models (LLMs) such as GPT-4 may offer a convenient yet risky alternative because they can produce inaccurate but convincing information. We therefore ask whether patients can trust clinical advice from AI-driven LLMs. We examined this question through a think-aloud observation in which a patient used GPT-4 for self-diagnosis while a doctor assessed its responses using professional expertise. We then conducted a semi-structured interview with the patient about their trust in the system. Our results show that patients may struggle to identify errors because they lack professional medical knowledge, even when GPT-4 provides advice that a doctor can recognize as false. Patients may develop some trust because GPT-4 explains its responses and acknowledges its limitations, but this trust remains uncertain because its advice can be unreliable. The doctor also reported that checking GPT-4's responses required more effort than making a diagnosis without it. Patients tend to trust doctors because educated and authorized professionals can provide effective guidance. This trust also develops through social connection, certification, institutional accountability, and professional rules. Doctors can adapt their questions, observe patients, and use different methods when patients cannot clearly describe their symptoms. An LLM, however, depends primarily on the information provided in a prompt and may overlook details that a doctor could identify during a clinical consultation. These findings raise questions about competence, responsibility, autonomy, and safety when LLMs are used for clinical advice.
What Shapes User Trust in ChatGPT? A Mixed-Methods Study of User Attributes, Trust Dimensions, Task Context, and Societal Perceptions among University Students
Measure Utility, Gain Trust: Practical Advice for XAI Researcher
School of Engineering and Applied Sciences Washington State University(华盛顿州立大学工程与应用科学学院);Data Sciences and Analytics Group Pacific Northwest National Laboratory(太平洋西北国家实验室数据科学与分析组);Cognitive Engineering Center Georgia Institute of Technology(佐治亚理工学院认知工程中心);Visual Analytics Group Pacific Northwest National Laboratory(太平洋西北国家实验室可视化分析组)
AI Ethics and Social Norms: Exploring ChatGPT's Capabilities From What to How
University of Toronto(多伦多大学);Drexel University(德雷塞尔大学);University of Siegen(锡根大学);Westphal College of Media Arts and Design(媒体艺术与设计西弗尔学院);Digital Media Department(数字媒体系);Information Systems(信息系统)