Recent studies have demonstrated promising performance of ChatGPT and GPT-4 on several medical domain tasks. However, none have assessed its performance using a large-scale real-world electronic health record database, nor have evaluated its utility in providing clinical diagnostic assistance for patients across a full range of disease presentation. We performed two analyses using ChatGPT and GPT-4, one to identify patients with specific medical diagnoses using a real-world large electronic health record database and the other, in providing diagnostic assistance to healthcare workers in the prospective evaluation of hypothetical patients. Our results show that GPT-4 across disease classification tasks with chain of thought and few-shot prompting can achieve performance as high as 96% F1 scores. For patient assessment, GPT-4 can accurately diagnose three out of four times. However, there were mentions of factually incorrect statements, overlooking crucial medical findings, recommendations for unnecessary investigations and overtreatment. These issues coupled with privacy concerns, make these models currently inadequate for real world clinical use. However, limited data and time needed for prompt engineering in comparison to configuration of conventional machine learning workflows highlight their potential for scalability across healthcare applications.
Conversational Assistants to support Heart Failure Patients: comparing a Neurosymbolic Architecture with ChatGPT
Department of Computer Science(计算机科学系);Department of Biomedical and Health Information Sciences(生物医学与健康信息科学系);Department of Medicine(医学系);Department of Communications(通讯系)
ChatGPT Chemistry Assistant for Text Mining and Prediction of MOF Synthesis
University of California, Berkeley(加州大学伯克利分校);Kavli Energy Nanoscience Institute, University of California, Berkeley(加州大学伯克利分校卡夫利能源纳米科学研究所);King Abdulaziz City for Science and Technology(阿卜杜勒阿齐兹国王科技城)
Deutsche Welle(德国之声);Universitat Politecnica de Valencia(瓦伦西亚理工大学);ABC News(美国广播公司新闻部);Bonn-Aachen International Center for Information Technology (b-it), University of Bonn(波恩大学波恩-亚琛国际信息技术中心);The Lamarr Institute for Machine Learning and Artificial Intelligence(拉马尔机器学习与人工智能研究所)
Personalized Large Language Model Assistant with Evolving Conditional Memory
The Hong Kong Polytechnic University(香港理工大学);INF Technology (Shanghai) co., Ltd.(INF科技(上海)有限公司);Institute of Artificial Intelligence, Soochow University, China(苏州大学人工智能研究院)
The potential of large language models for improving probability learning: A study on ChatGPT3.5 and first-year computer engineering students
Rey Juan Carlos University(胡安·卡洛斯国王大学);European Commission, Joint Research Centre(欧盟委员会联合研究中心);CETINIA: Research Centre for Intelligent Information Technologies(CETINIA智能信息技术研究中心)
Towards conversational assistants for health applications: using ChatGPT to generate conversations about heart failure
Department of Computer Science(计算机科学系);Department of Biomedical and Health Information Sciences(生物医学与健康信息科学系);Department of Medicine(医学系);Department of Communications(通讯系)
Who is ChatGPT? Benchmarking LLMs' Psychological Portrayal Using PsychoBench
The Chinese University of Hong Kong(香港中文大学);Tianjin Medical University(天津医科大学);Tencent AI Lab(腾讯AI实验室);The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))
When "A Helpful Assistant" Is Not Really Helpful: Personas in System Prompts Do Not Improve Performances of Large Language Models
Carnegie Mellon University(卡内基梅隆大学);Stanford University(斯坦福大学);LG AI Research(LG人工智能研究院);University of Illinois Chicago(芝加哥大学伊利诺伊分校);University of Michigan(密歇根大学)
University of Oxford(牛津大学);University of Vienna(维也纳大学);Institute for Advanced Study(普林斯顿高等研究院);University of Cambridge(剑桥大学);Vienna University of Technology(维也纳工业大学);VERSES Research Lab(VERSES研究实验室)