The emergence of large language models (LLMs) has significantly influenced numerous fields, including healthcare, by enhancing the capabilities of automated systems to process and generate human-like text. However, despite their advancements, the reliability and accuracy of LLMs in medical contexts remain critical concerns. Current evaluation methods often lack robustness and fail to provide a comprehensive assessment of LLM performance, leading to potential risks in clinical settings. In this work, we propose Med-CoDE, a specifically designed evaluation framework for medical LLMs to address these challenges. The framework leverages a critique-based approach to quantitatively measure the degree of disagreement between model-generated responses and established medical ground truths. This framework captures both accuracy and reliability in medical settings. The proposed evaluation framework aims to fill the existing gap in LLM assessment by offering a systematic method to evaluate the quality and trustworthiness of medical LLMs. Through extensive experiments and case studies, we illustrate the practicality of our framework in providing a comprehensive and reliable evaluation of medical LLMs.
MCQG-SRefine: Multiple Choice Question Generation and Evaluation with Iterative Self-Critique, Correction, and Comparison Feedback
University of Massachusetts, Amherst（马萨诸塞大学阿默斯特分校）；University of Minnesota（明尼苏达大学）；University of Massachusetts, Lowell（马萨诸塞大学洛厄尔分校）；UMass Chan Medical School（马萨诸塞大学陈医学院）
LLM-Based Scientific Peer Review: Methods, Benchmarks, and Reliability Challenges
L3S Research Center, Leibniz University Hannover（莱布尼茨汉诺威大学L3S研究中心）；Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School（布伦瑞克工业大学与汉诺威医学院彼得·L·赖歇茨医学信息学研究所）；Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed)（下萨克森州医学人工智能与因果方法中心（CAIMed））
LLM Self-Correction with DeCRIM: Decompose, Critique, and Refine for Enhanced Following of Instructions with Multiple Constraints
Amazon AGI Foundations（亚马逊AGI基础研究院）；Télécom Paris（巴黎电信学院）；Institut Polytechnique de Paris（巴黎综合理工学院）；UNC Chapel Hill（北卡罗来纳大学教堂山分校）；University of California, Los Angeles（加利福尼亚大学洛杉矶分校）
Training-Free Cultural Alignment of Large Language Models via Persona Disagreement
Faculty of Information and Technology, University of Science, Vietnam National University（信息与技术学院，科学大学，越南国家大学）；Department of Computer Science, University of Warwick（计算机科学系，沃里克大学）；School of Computing, Engineering and Digital Technologies, Teesside University（计算、工程和数字技术学院，泰赛德大学）
Symbolic Intermediaries as a Linguistic-Numerical Interface for LLM-Driven Geometric Reasoning
Idiap Research Institute（Idiap研究 institute）；École Polytechnique Fédérale de Lausanne（瑞士联邦理工学院）；Honda Research Institute Europe（本田欧洲研究院）；Department of Computer Science, University of Manchester（曼彻斯特大学计算机科学系）；National Biomarker Centre, CRUK-MI, University of Manchester（曼彻斯特大学国家生物标记中心）
College of Computing, UNC Charlotte（北卡罗来纳大学夏洛特分校计算学院）；School of Data Science, UNC Charlotte（北卡罗来纳大学夏洛特分校数据科学学院）；Departments of Philosophy & Medical Education, U. of Central Florida（中佛罗里达大学哲学与医学教育系）
Using natural language processing tools, we investigate the differences of recommendations in medical guidelines for the same decision problem -- breast cancer screening. We show that these differences arise from knowledge brought to the problem by different medical societies, as reflected in the conceptual vocabularies used by the different groups of authors.The computational models we build and analyze agree with the near-peer epistemic model of expert disagreement proposed by Garbayo. Even though the article is a case study focused on one set of guidelines, the proposed methodology is broad…
Can Large Language Models Self-Correct in Medical Question Answering? An Exploratory Study
Department of Electrical and Computer Engineering, University of Minnesota（明尼苏达大学电气与计算机工程系）；Division of Computational Health Sciences, Department of Surgery, University of Minnesota（明尼苏达大学外科学系计算健康科学部）；Department of Software Engineering, University of St. Thomas（圣托马斯大学软件工程系）
