Evaluation of ChatGPT Family of Models for Biomedical Reasoning and Classification
Mass General Brigham, Harvard Medical School（麻省总医院布莱根，哈佛医学院）；Brigham and Women’s Hospital/Dana-Farber Cancer Institute（布莱根妇女医院/丹娜-法伯癌症研究所）；Technische Universität Darmstadt（达姆施塔特工业大学）；Maastricht University（马斯特里赫特大学）
Shan Chen, Yingya Li, Sheng Lu, Hoang Van, Hugo JWL Aerts, Guergana K. Savova, Danielle S. Bitterman
备注：
28 pages, 2 tables and 4 figures. Submitting for review
Recent advances in large language models (LLMs) have shown impressive ability in biomedical question-answering, but have not been adequately investigated for more specific biomedical applications. This study investigates the performance of LLMs such as the ChatGPT family of models (GPT-3.5s, GPT-4) in biomedical tasks beyond question-answering. Because no patient data can be passed to the OpenAI API public interface, we evaluated model performance with over 10000 samples as proxies for two fundamental tasks in the clinical domain - classification and reasoning. The first task is classifying whether statements of clinical and policy recommendations in scientific literature constitute health advice. The second task is causal relation detection from the biomedical literature. We compared LLMs with simpler models, such as bag-of-words (BoW) with logistic regression, and fine-tuned BioBERT models. Despite the excitement around viral ChatGPT, we found that fine-tuning for two fundamental NLP tasks remained the best strategy. The simple BoW model performed on par with the most complex LLM prompting. Prompt engineering required significant investment.
Bioinformatics and Biomedical Informatics with ChatGPT: Year One Review
West Virginia University（西弗吉尼亚大学）；University of Nebraska-Lincoln（内布拉斯加大学林肯分校）；Arizona State University（亚利桑那州立大学）；Biodesign Institute, Arizona State University（亚利桑那州立大学生物设计研究所）
From Knowledge Generation to Knowledge Verification: Examining the BioMedical Generative Capabilities of ChatGPT
MGEN -- College of Engineering, Northeastern University Miami（东北大学迈阿密分校MGEN工程学院）；University of Vermont（佛蒙特大学）；AGH, University of Krakow（克拉科夫AGH大学）；Howard University（霍华德大学）
ChatABL: Abductive Learning via Natural Language Interaction with ChatGPT
Northwestern Polytechnical University（西北工业大学）；The University of Georgia（佐治亚大学）；University of Electronic Science and Technology of China（电子科技大学）；Massachusetts General Hospital（麻省总医院）；Harvard Medical School（哈佛医学院）；The University of Texas at Arlington（德克萨斯大学阿灵顿分校）；ShanghaiTech University（上海科技大学）；Shanghai United Imaging Intelligence Co., Ltd.（上海联影智能医疗科技有限公司）
An LLM-based Knowledge Synthesis and Scientific Reasoning Framework for Biomedical Discovery
Idiap Research Institute（伊迪亚普研究所）；National Biomarker Centre（国家生物标志物中心）；CRUK-MI（英国癌症研究中心曼彻斯特研究所）；Univ. of Manchester（曼彻斯特大学）；Department of Computer Science, Univ. of Manchester（曼彻斯特大学计算机科学系）
The potential of large language models for improving probability learning: A study on ChatGPT3.5 and first-year computer engineering students
Rey Juan Carlos University（胡安·卡洛斯国王大学）；European Commission, Joint Research Centre（欧盟委员会联合研究中心）；CETINIA: Research Centre for Intelligent Information Technologies（CETINIA智能信息技术研究中心）
MedAgentGym: A Scalable Agentic Training Environment for Code-Centric Reasoning in Biomedical Data Science
Emory University（埃默里大学）；Georgia Institute of Technology（佐治亚理工学院）；University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Yale University（耶鲁大学）；UT Southwestern Medical Center（德克萨斯大学西南医学中心）；University of Washington（华盛顿大学）
MedHopQA: A Disease-Centered Multi-Hop Reasoning Benchmark and Evaluation Framework for LLM-Based Biomedical Question Answering
National Library of Medicine, Division of Intramural Research（国家医学图书馆，院内研究部）；University of Illinois at Urbana-Champaign, Department of Computer Science（伊利诺伊大学厄巴纳-香槟分校计算机科学系）；University of Michigan Medical School（密歇根大学医学院）
BiomedSQL: Text-to-SQL for Scientific Reasoning on Biomedical Knowledge Bases
Center for Alzheimer’s and Related Dementias, NIA, NIH（阿尔茨海默病及相关痴呆症研究中心，国家老龄化研究所，国立卫生研究院）；DataTecnica LLC（DataTecnica公司）；Johns Hopkins University（约翰霍普金斯大学）；Laboratory of Neurogenetics, NIA, NIH（神经遗传学实验室，国家老龄化研究所，国立卫生研究院）
Can ChatGPT Overcome Behavioral Biases in the Financial Sector? Classify-and-Rethink: Multi-Step Zero-Shot Reasoning in the Gold Investment
Hong Kong University of Science and Technology（香港科技大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；E Fund Management Co., Ltd（易方达基金管理有限公司）
LLMs are not Zero-Shot Reasoners for Biomedical Information Extraction
ASUS Intelligent Cloud Services (AICS)（ASUS智能云服务）；Imperial College London（伦敦帝国学院）；University of Manchester（曼彻斯特大学）；National University of Singapore（新加坡国立大学）
