Clinical Validation of Medical-based Large Language Model Chatbots on Ophthalmic Patient Queries with LLM-based Evaluation
Singapore National Eye Centre（新加坡国家眼科中心）；Singapore Eye Research Institute（新加坡眼科研究所）；Singapore Health Services（新加坡保健服务集团）；Medical University of Vienna（维也纳医科大学）；The Hospital at Maayo（马约医院）；Duke-NUS Medical School（杜克-新加坡国立大学医学院）；London School of Hygiene and Tropical Medicine（伦敦卫生与热带医学院）；Chang Gung Memorial Hospital（长庚纪念医院）；Taipei Medical University（台北医学大学）；National Health Research Institutes（国家卫生研究院）；Nanyang Technological University（南洋理工大学）
Ting Fang Tan, Kabilan Elangovan, Andreas Pollreisz, Kevin Bryan Dy, Wei Yan Ng, Joy Le Yi Wong, Jin Liyuan, Chrystie Quek Wan Ning, Ashley Shuen Ying Hong, Aru… 展开作者
Ting Fang Tan, Kabilan Elangovan, Andreas Pollreisz, Kevin Bryan Dy, Wei Yan Ng, Joy Le Yi Wong, Jin Liyuan, Chrystie Quek Wan Ning, Ashley Shuen Ying Hong, Arun James Thirunavukarasu, Shelley Yin-His Chang, Jie Yao, Dylan Hong, Wang Zhaoran, Amrita Gupta, Daniel SW Ting
Domain specific large language models are increasingly used to support patient education, triage, and clinical decision making in ophthalmology, making rigorous evaluation essential to ensure safety and accuracy. This study evaluated four small medical LLMs Meerkat-7B, BioMistral-7B, OpenBioLLM-8B, and MedLLaMA3-v20 in answering ophthalmology related patient queries and assessed the feasibility of LLM based evaluation against clinician grading. In this cross sectional study, 180 ophthalmology patient queries were answered by each model, generating 2160 responses. Models were selected for parameter sizes under 10 billion to enable resource efficient deployment. Responses were evaluated by three ophthalmologists of differing seniority and by GPT-4-Turbo using the S.C.O.R.E. framework assessing safety, consensus and context, objectivity, reproducibility, and explainability, with ratings assigned on a five point Likert scale. Agreement between LLM and clinician grading was assessed using Spearman rank correlation, Kendall tau statistics, and kernel density estimate analyses. Meerkat-7B achieved the highest performance with mean scores of 3.44 from Senior Consultants, 4.08 from Consultants, and 4.18 from Residents. MedLLaMA3-v20 performed poorest, with 25.5 percent of responses containing hallucinations or clinically misleading content, including fabricated terminology. GPT-4-Turbo grading showed strong alignment with clinician assessments overall, with Spearman rho of 0.80 and Kendall tau of 0.67, though Senior Consultants graded more conservatively. Overall, medical LLMs demonstrated potential for safe ophthalmic question answering, but gaps remained in clinical depth and consensus, supporting the feasibility of LLM based evaluation for large scale benchmarking and the need for hybrid automated and clinician review frameworks to guide safe clinical deployment.
Visual Question Answering in Ophthalmology: A Progressive and Practical Perspective
School of Optometry, The Hong Kong Polytechnic University（香港理工大学视光学院）；Research Centre for SHARP Vision (RCSV), The Hong Kong Polytechnic University（香港理工大学SHARP视觉研究中心）；Centre for Eye and Vision Research (CEVR)（眼科及视觉科学研究中心）
Developing and evaluating a chatbot to support maternal health care
Carnegie Mellon University（卡内基梅隆大学）；Population Council Institute（人口理事会研究所）；Sitaram Bhartia Institute of Science and Research（西塔拉姆·巴提亚科学与研究研究所）；Nivi, Inc.（Nivi公司）
Fine-tuning Large Language Model (LLM) Artificial Intelligence Chatbots in Ophthalmology and LLM-based evaluation using GPT-4
Singapore National Eye Centre（新加坡全国眼科中心）；Singapore Eye Research Institute（新加坡眼科研究所）；Singapore Health Services（新加坡保健服务）；Duke-NUS Medical School（杜克-新加坡国立大学医学院）；Singapore General Hospital（新加坡中央医院）
AGENTiGraph: An Interactive Knowledge Graph Platform for LLM-based Chatbots Utilizing Private Data
The University of Tokyo（东京大学）；Universität Bielefeld（比勒费尔德大学）；Duke-NUS Medical School（杜克-新加坡国立大学医学院）；Universidad de Alicante（阿利坎特大学）；Yale University（耶鲁大学）；Xiamen University（厦门大学）；Weill Cornell Medicine（威尔康奈尔医学院）；Henan University（河南大学）
Making Them Ask and Answer: Jailbreaking Large Language Models in Few Queries via Disguise and Reconstruction
Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络空间安全学院）；RealAI；Tsinghua University（清华大学）
Understanding Learner-LLM Chatbot Interactions and the Impact of Prompting Guidelines
Università degli Studi di Milano Bicocca（米兰比可卡大学）；Universitat Pompeu Fabra（庞培法布拉大学）；University of Regensburg（雷根斯堡大学）；Politecnico di Milano（米兰理工大学）；Istituto per le Tecnologie Didattiche (ITD-CNR)（国家研究委员会教育技术研究所）；Università degli Studi di Milano（米兰大学）；Institute of Science and Technology of Cognition (ISTC-CNR)（国家研究委员会认知科学与技术研究所）
An Approach for Auto Generation of Labeling Functions for Software Engineering Chatbots
Data-driven Analysis of Software (DAS) Lab Department of Computer Science \& Software Engineering Concordia University Montréal QC Canada；Department of Electrical \& Software Engineering University of Calgary Calgary AB Canada；Concordia University；University of Calgary
Why So Toxic? Measuring and Triggering Toxic Behavior in Open-Domain Chatbots
CISPA Helmholtz Center for Information Security（CISPA亥姆霍兹信息安全中心）；Binghamton University（宾汉姆顿大学）；University College London（伦敦大学学院）；Boston University（波士顿大学）；TU Delft（代尔夫特理工大学）
Quriosity: Analyzing Human Questioning Behavior and Causal Inquiry through Curiosity-Driven Queries
EPFL（苏黎世联邦理工学院）；ETH Zürich（苏黎世联邦理工学院）；University of Washington（华盛顿大学）；University of Toronto（多伦多大学）；Vector Institute（向量研究所）；University of Michigan（密歇根大学）；MPI for Intelligent Systems, Tübingen, Germany（图宾根智能系统研究所）
Place Matters: Comparing LLM Hallucination Rates for Place-Based Legal Queries
School of Computing and Information Systems, The University of Melbourne, Australia（计算与信息系统学院，墨尔本大学，澳大利亚）；The Centre for Artificial Intelligence and Digital Ethics（人工智能与数字伦理中心）；Melbourne Law School, The University of Melbourne, Australia（墨尔本法学院，墨尔本大学，澳大利亚）
Designing an Evaluation Framework for Large Language Models in Astronomy Research
Space Telescope Science Institute（空间望远镜科学研究所）；Johns Hopkins University（约翰斯·霍普金斯大学）；Oregon State University（俄勒冈州立大学）；Loyola University Maryland（马里兰州洛约拉大学）；Stanford University（斯坦福大学）；European Space Astronomy Centre（欧洲空间天文中心）；Australian National University（澳大利亚国立大学）；Columbia University（哥伦比亚大学）
Towards Enhancing Linked Data Retrieval in Conversational UIs using Large Language Models
School of Computer Science and Informatics, Cardiff University（卡迪夫大学计算机科学与信息学院）；College of Computing and Informatics, Saudi Electronic University（沙特电子大学计算与信息学院）；School of Biosciences, Cardiff University（卡迪夫大学生物科学学院）
