This study investigates GPT-4's assessment of its performance in healthcare applications. A simple prompting technique was used to prompt the LLM with questions taken from the United States Medical Licensing Examination (USMLE) questionnaire and it was tasked to evaluate its confidence score before posing the question and after asking the question. The questionnaire was categorized into two groups-questions with feedback (WF) and questions with no feedback(NF) post-question. The model was asked to provide absolute and relative confidence scores before and after each question. The experimental findings were analyzed using statistical tools to study the variability of confidence in WF and NF groups. Additionally, a sequential analysis was conducted to observe the performance variation for the WF and NF groups. Results indicate that feedback influences relative confidence but doesn't consistently increase or decrease it. Understanding the performance of LLM is paramount in exploring its utility in sensitive areas like healthcare. This study contributes to the ongoing discourse on the reliability of AI, particularly of LLMs like GPT-4, within healthcare, offering insights into how feedback mechanisms might be optimized to enhance AI-assisted medical education and decision support.
Can large language models reason about medical questions?
Technical University of Denmark（丹麦技术大学）；FindZebra ApS（FindZebra 公司）；Rigshospitalet, Copenhagen University Hospital（哥本哈根大学医院 Rigshospitalet）；University of Copenhagen（哥本哈根大学）
DERA: Enhancing Large Language Model Completions with Dialog-Enabled Resolving Agents
Curai Health（Curai健康）
大语言模型（LLM）已经成为许多自然语言理解任务的有价值的工具。在医疗保健等安全关键型应用中，这些模型的实用性取决于它们生成真实准确和完整的输出的能力。在这项工作中，我们提出了对话使能解析代理（DERA）。DERA是一个范例，通过LLM的会话能力的增加而成为可能，即GPT-4。它为模型提供了一个简单的、可解释的论坛，以交流反馈并迭代地改进输出。我们框架我们的对话作为两个代理类型之间的讨论-一个研究员，谁处理信息，并确定关键的问题组件，和一个决策者，谁拥有自主权，以整合研究员的信息，并作出判断的最终输出。我们测试DERA对三个临床重点任务。对于医疗会话摘要和护理计划生成，DERA在人类专家偏好评估和定量指标方面都显示出比基本GPT-4性能的显着改进。在一项新的发现中，我们还表明，GPT-4在MedQA问答（QA）数据集（Jin et al. 2021，USMLE）的开放式版本上的性能（70%）远高于及格水平（60%），DERA表现出类似的性能。我们在https://github.com/curai/curai-research/tree/main/DERA上发布开放式MEDQA数据集。
MCQG-SRefine: Multiple Choice Question Generation and Evaluation with Iterative Self-Critique, Correction, and Comparison Feedback
University of Massachusetts, Amherst（马萨诸塞大学阿默斯特分校）；University of Minnesota（明尼苏达大学）；University of Massachusetts, Lowell（马萨诸塞大学洛厄尔分校）；UMass Chan Medical School（马萨诸塞大学陈医学院）
Evaluating GPT- and Reasoning-based Large Language Models on Physics Olympiad Problems: Surpassing Human Performance and Implications for Educational Assessment
Leibniz Institute for Science and Mathematics Education（莱布尼茨科学与数学教育研究所）；Free University of Berlin（柏林自由大学）；Ruprecht Karl University of Heidelberg（鲁普莱希特-卡尔斯海德堡大学）；Ludwigsburg University of Education（路德维希堡教育大学）
The use of GPT-4o and Other Large Language Models for the Improvement and Design of Self-Assessment Scales for Measurement of Interpersonal Communication Skills
University of Zagreb（萨格勒布大学）；Faculty of Organization and Informatics（组织与信息学院）
DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Models
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Stanford University（斯坦福大学）；University of California, Berkeley（加州大学伯克利分校）；Center for AI Safety（AI安全中心）；Microsoft Corporation（微软公司）；The Chinese University of Hong Kong（香港中文大学）
Large Language Model-Driven Dynamic Assessment of Grammatical Accuracy in English Language Learner Writing
National Institute of Standards（国家标准 institute）；Department of Physics, Colorado State University（物理系，科罗拉多州立大学）；Electrical Engineering Department, University of Colorado（电气工程系，科罗拉多大学）
Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiency
University of Guelph（圭尔夫大学）；University of North Texas（北德克萨斯大学）；Iowa State University（爱荷华州立大学）；Zayed University（扎耶德大学）；University of Texas at Dallas（德克萨斯大学达拉斯分校）；National University of Computer & Emerging Sciences（国立计算机与新兴科学大学）
Mind meets machine: Unravelling GPT-4's cognitive psychology
Nowrosjee Wadia College（诺罗斯吉·瓦迪亚学院）；Indian Institute of Tropical Meteorology（印度热带气象研究所）；Defence Institute Of Advanced Technology（国防先进技术研究所）；Queen Mary University of London（伦敦玛丽女王大学）
Hey GPT-OSS, Looks Like You Got It -- Now Walk Me Through It! An Assessment of the Reasoning Language Models Chain of Thought Mechanism for Digital Forensics
Chair for Cybersecurity, University of Augsburg, Augsburg, Germany（网络安全系，乌尔姆大学，乌尔姆，德国）
Towards Safer Chatbots: Automated Policy Compliance Evaluation of Custom GPTs
Information Processing and Telecommunications Center Universidad Politécnica de Madrid ETSI Telecomunicación（信息处理与电信中心西班牙马德里理工大学电信工程学院）；King’s College London（伦敦国王学院）；INGENIO (CSIC-Universitat Politècnica de València)（INGENIO（西班牙国家研究委员会-瓦伦西亚理工大学））
GPT as Knowledge Worker: A Zero-Shot Evaluation of (AI)CPA Capabilities
Illinois Tech - Chicago Kent College of Law（伊利诺伊理工大学芝加哥肯特法学院）；Bucerius Law School（布塞留斯法学院）；CodeX - The Stanford Center for Legal Informatics（CodeX——斯坦福法律信息学中心）
COMPARE: Clinical Optimization with Modular Planning and Assessment via RAG-Enhanced AI-OCT: Superior Decision Support for Percutaneous Coronary Intervention Compared to ChatGPT-5 and Junior Operators
Tangdu Hospital, The Fourth Military Medical University（空军军医大学唐都医院）；Research Institute of Tsinghua University in Shenzhen（深圳清华大学研究院）；The Second Affiliated Hospital, Air Force Medical University（空军军医大学第二附属医院）
Do Benchmarks Underestimate LLM Performance? Evaluating Hallucination Detection With LLM-First Human-Adjudicated Assessment
Department of Computer Engineering, Hacettepe University（哈切塔佩大学计算机工程系）；Department of Computer Engineering, Ankara University（安卡拉大学计算机工程系）；Zephlen AI and Information Technologies Inc.（泽夫伦人工智能与信息技术公司）
Large Language Models for Mental Health Diagnostic Assessments: Exploring The Potential of Large Language Models for Assisting with Mental Health Diagnostic Assessments -- The Depression and Anxiety Case
Indian Institute of Research and Science, Bhopal（博帕尔印度研究与科学学院）；Indian Institute of Technology, Kharagpur（克勒格布尔印度理工学院）
Performance of ChatGPT-3.5 and GPT-4 on the United States Medical Licensing Examination With and Without Distractions
Magee-Womens Research Institute（马吉妇女研究所）；University of Pittsburgh School of Medicine（匹兹堡大学医学院）；Chaim Sheba Medical Center（哈伊姆·谢巴医疗中心）；Sackler Faculty of Medicine, Tel Aviv University（特拉维夫大学萨克勒医学院）；Ariel University（阿里尔大学）
Large Language Models (GPT) for automating feedback on programming assignments
Institute of Information Technology, Warsaw University of Life Sciences（华沙生命科学大学信息技术研究所）；Penn Center for Learning Analytics, University of Pennsylvania（宾夕法尼亚大学宾夕法尼亚学习分析中心）
Clinical Camel: An Open Expert-Level Medical Language Model with Dialogue-Based Knowledge Encoding
Vector Institute for Artificial Intelligence（矢量人工智能研究所）；University of Toronto（多伦多大学）；Peter Munk Cardiac Centre, University Health Network（大学健康网络彼得·蒙克心脏中心）；McGill University（麦吉尔大学）；University Health Network（大学健康网络）
