Fine-tuning Large Language Model (LLM) Artificial Intelligence Chatbots in Ophthalmology and LLM-based evaluation using GPT-4
Singapore National Eye Centre（新加坡全国眼科中心）；Singapore Eye Research Institute（新加坡眼科研究所）；Singapore Health Services（新加坡保健服务）；Duke-NUS Medical School（杜克-新加坡国立大学医学院）；Singapore General Hospital（新加坡中央医院）
Ting Fang Tan, Kabilan Elangovan, Liyuan Jin, Yao Jie, Li Yong, Joshua Lim, Stanley Poh, Wei Yan Ng, Daniel Lim, Yuhe Ke, Nan Liu, Daniel Shu Wei Ting
Purpose: To assess the alignment of GPT-4-based evaluation to human clinician experts, for the evaluation of responses to ophthalmology-related patient queries generated by fine-tuned LLM chatbots. Methods: 400 ophthalmology questions and paired answers were created by ophthalmologists to represent commonly asked patient questions, divided into fine-tuning (368; 92%), and testing (40; 8%). We find-tuned 5 different LLMs, including LLAMA2-7b, LLAMA2-7b-Chat, LLAMA2-13b, and LLAMA2-13b-Chat. For the testing dataset, additional 8 glaucoma QnA pairs were included. 200 responses to the testing dataset were generated by 5 fine-tuned LLMs for evaluation. A customized clinical evaluation rubric was used to guide GPT-4 evaluation, grounded on clinical accuracy, relevance, patient safety, and ease of understanding. GPT-4 evaluation was then compared against ranking by 5 clinicians for clinical alignment. Results: Among all fine-tuned LLMs, GPT-3.5 scored the highest (87.1%), followed by LLAMA2-13b (80.9%), LLAMA2-13b-chat (75.5%), LLAMA2-7b-Chat (70%) and LLAMA2-7b (68.8%) based on the GPT-4 evaluation. GPT-4 evaluation demonstrated significant agreement with human clinician rankings, with Spearman and Kendall Tau correlation coefficients of 0.90 and 0.80 respectively; while correlation based on Cohen Kappa was more modest at 0.50. Notably, qualitative analysis and the glaucoma sub-analysis revealed clinical inaccuracies in the LLM-generated responses, which were appropriately identified by the GPT-4 evaluation. Conclusion: The notable clinical alignment of GPT-4 evaluation highlighted its potential to streamline the clinical evaluation of LLM chatbot responses to healthcare-related queries. By complementing the existing clinician-dependent manual grading, this efficient and automated evaluation could assist the validation of future developments in LLM applications for healthcare.
Mayo Clinic（梅奥诊所）；the First Affiliated Hospital of Nanchang University（南昌大学第一附属医院）；University of Georgia（佐治亚大学）；University of Electronic Science and Technology of China（电子科技大学）；Eye Institute of Xiamen University（厦门大学眼科研究所）
Clinical Validation of Medical-based Large Language Model Chatbots on Ophthalmic Patient Queries with LLM-based Evaluation
Singapore National Eye Centre（新加坡国家眼科中心）；Singapore Eye Research Institute（新加坡眼科研究所）；Singapore Health Services（新加坡保健服务集团）；Medical University of Vienna（维也纳医科大学）；The Hospital at Maayo（马约医院）；Duke-NUS Medical School（杜克-新加坡国立大学医学院）；London School of Hygiene and Tropical Medicine（伦敦卫生与热带医学院）；Chang Gung Memorial Hospital（长庚纪念医院）；Taipei Medical University（台北医学大学）；National Health Research Institutes（国家卫生研究院）；Nanyang Technological University（南洋理工大学）
Towards Safer Chatbots: Automated Policy Compliance Evaluation of Custom GPTs
Information Processing and Telecommunications Center Universidad Politécnica de Madrid ETSI Telecomunicación（信息处理与电信中心西班牙马德里理工大学电信工程学院）；King’s College London（伦敦国王学院）；INGENIO (CSIC-Universitat Politècnica de València)（INGENIO（西班牙国家研究委员会-瓦伦西亚理工大学））
Can OpenAI o1 Reason Well in Ophthalmology? A 6,990-Question Head-to-Head Evaluation Study
National University of Singapore（新加坡国立大学）；Singapore Eye Research Institute（新加坡眼科研究所）；Yale University（耶鲁大学）；Cleveland Clinic（克利夫兰诊所）；Zhejiang University（浙江大学）
BEnchmarking LLMs for Ophthalmology (BELO) for Ophthalmological Knowledge and Reasoning
National University of Singapore（新加坡国立大学）；Yale University（耶鲁大学）；Singapore Eye Research Institute（新加坡眼科研究所）；Singapore National Eye Centre（新加坡全国眼科中心）；Massachusetts Eye and Ear（麻省眼耳医院）；Harvard Medical School（哈佛医学院）；Nanyang Technological University（南洋理工大学）；Singapore University of Technology and Design（新加坡科技设计大学）
ChatThero: An LLM-Supported Chatbot for Behavior Change and Therapeutic Support in Addiction Recovery
Center for Healthcare Organization and Implementation Research, VA Bedford Health Care（VA Bedford Health Care 医疗组织与实施研究中心）；Manning College of Information and Computer Sciences, University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校信息与计算机科学学院）；Miner School of Computer and Information Sciences, University of Massachusetts Lowell（马萨诸塞大学洛厄尔分校计算机与信息科学学院）
Holistic Evaluation of GPT-4V for Biomedical Imaging
University of Georgia（佐治亚大学）；Beijing Jiaotong University（北京交通大学）；Northwestern Polytechnical University（西北工业大学）；University of Texas at Arlington（德克萨斯大学阿灵顿分校）；University of Electronic Science and Technology of China（电子科技大学）
The Machine Psychology of Cooperation: Can GPT models operationalise prompts for altruism, cooperation, competitiveness and selfishness in economic games?
University College London（伦敦大学学院）；Middlesex University（密德萨斯大学）
Building Trust in Mental Health Chatbots: Safety Metrics and LLM-Based Evaluation Tools
Sue & Bill Gross School of Nursing, University of California, Irvine（加州大学欧文分校苏与比尔·格罗斯护理学院）；Department of Computer Sciences, University of California, Irvine（加州大学欧文分校计算机科学系）；School of Medicine, University of California, Irvine（加州大学欧文分校医学院）；Department of Psychological Science, University of California, Irvine（加州大学欧文分校心理科学系）；Chicago School of Professional Psychology（芝加哥专业心理学院）；HealthUnity
Benchmarking Large Multimodal Models for Ophthalmic Visual Question Answering with OphthalWeChat
The Hong Kong Polytechnic University（香港理工大学）；Swiss Federal Institute of Technology Lausanne (EPFL)（洛桑联邦理工学院）；PolyU-Wuhan Technology and Innovation Research Institute（香港理工大学武汉技术创新研究院）；University of Liverpool（利物浦大学）；Centre for Eye and Vision Research (CEVR)（眼科视觉研究中心）
Mapping Caregiver Needs to AI Chatbot Design: Strengths and Gaps in Mental Health Support for Alzheimer's and Dementia Caregivers
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Indiana University Indianapolis（印第安纳大学印第安纳波利斯分校）；University of Massachusetts Amherst（马萨诸塞大学阿姆赫斯特分校）
Do AI chatbots find what experts would? Effects of model, user role, and sample size on study retrieval for medical questions
National Institute on Drug Abuse（国家药物滥用研究所）；National Institutes of Health（美国国立卫生研究院）；National Library of Medicine（国家医学图书馆）；School of Information Sciences, University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校信息科学学院）
Comparative Analysis of ChatGPT, GPT-4, and Microsoft Bing Chatbots for GRE Test
Thornton Tomasetti, Inc.（Thornton Tomasetti 公司）；Yarmouk University（耶尔穆克大学）；Ministry of Municipal Affairs（市政事务部）；Jordan University of Science and Technology（约旦科技大学）
Performance of a large language model-Artificial Intelligence based chatbot for counseling patients with sexually transmitted infections and genital diseases
University College of Medical Sciences and Guru Teg Bahadur Hospital（大学医学院与古鲁·特格·巴哈杜尔医院）；HeHealth Inc.（HeHealth公司）
Solving the multiplication problem of a large language model system using a graph-based method
Firat University（菲拉特大学）；Erzurum Technical University（埃尔祖鲁姆技术大学）；University of Southern Queensland（南昆士兰大学）；National Heart Centre Singapore（新加坡国家心脏中心）；Duke-NUS Medical School（杜克-新加坡国立大学医学院）；University of New England（新英格兰大学）；University of Technology Sydney（悉尼科技大学）
Potential of Multimodal Large Language Models for Data Mining of Medical Images and Free-text Reports
Institute of Medical Research, Northwestern Polytechnical University（西北工业大学医学研究院）；School of Computing, University of Georgia（佐治亚大学计算学院）；School of Life Science and Technology, University of Electronic Science and Technology of China（电子科技大学生命科学与技术学院）；School of Automation, Northwestern Polytechnical University（西北工业大学自动化学院）；School of Electronics and Information, Northwestern Polytechnical University（西北工业大学电子信息学院）；School of Biomedical Engineering, ShanghaiTech University（上海科技大学生物医学工程学院）；Lingang Laboratory（临港实验室）
