Evaluating Large Language Models for Public Health Classification and Extraction Tasks
UK Health Security Agency(英国卫生安全局)
Joshua Harris, Timothy Laurence, Leo Loman, Fan Grayson, Toby Nonnenmacher, Harry Long, Loes WalsGriffith, Amy Douglas, Holly Fountain, Stelios Georgiou, Jo Har… 展开作者
Joshua Harris, Timothy Laurence, Leo Loman, Fan Grayson, Toby Nonnenmacher, Harry Long, Loes WalsGriffith, Amy Douglas, Holly Fountain, Stelios Georgiou, Jo Hardstaff, Kathryn Hopkins, Y-Ling Chi, Galena Kuyumdzhieva, Lesley Larkin, Samuel Collins, Hamish Mohammed, Thomas Finnie, Luke Hounsome, Michael Borowitz, Steven Riley
备注:
36 pages. Feedback and comments are highly appreciated
Advances in Large Language Models (LLMs) have led to significant interest in their potential to support human experts across a range of domains, including public health. In this work we present automated evaluations of LLMs for public health tasks involving the classification and extraction of free text. We combine six externally annotated datasets with seven new internally annotated datasets to evaluate LLMs for processing text related to: health burden, epidemiological risk factors, and public health interventions. We evaluate eleven open-weight LLMs (7-123 billion parameters) across all tasks using zero-shot in-context learning. We find that Llama-3.3-70B-Instruct is the highest performing model, achieving the best results on 8/16 tasks (using micro-F1 scores). We see significant variation across tasks with all open-weight LLMs scoring below 60% micro-F1 on some challenging tasks, such as Contact Classification, while all LLMs achieve greater than 80% micro-F1 on others, such as GI Illness Classification. For a subset of 11 tasks, we also evaluate three GPT-4 and GPT-4o series models and find comparable results to Llama-3.3-70B-Instruct. Overall, based on these initial results we find promising signs that LLMs may be useful tools for public health experts to extract information from a wide variety of free text sources, and support public health surveillance, research, and interventions.
CHBench: A Chinese Dataset for Evaluating Health in Large Language Models
School of Artificial intelligence, Jilin University(吉林大学人工智能学院);Engineering Research Center of Knowledge-Driven Human-Machine Intelligence, Jilin University(吉林大学知识驱动人机智能工程研究中心);International Center of Future Science, Jilin University(吉林大学未来科学国际中心)
RephQA: Evaluating Readability of Large Language Models in Public Health Question Answering
Yale University, Department of Computer Science(耶鲁大学计算机科学系);Yale University, School of Nursing(耶鲁大学护理学院);Yale University, School of Public Health(耶鲁大学公共卫生学院);Northeastern University(东北大学)
A Dataset for Evaluating LLM-based Evaluation Functions for Research Question Extraction Task
Japan Advanced Institute of Science and Technology(日本先进科学技术学院);National Institute of Advanced Industrial Science and Technology(日本国立先进工业科学技术研究所);OMRON SINIC X Corporation(欧姆龙SINIC X公司)
Evaluating LLM Abilities to Understand Tabular Electronic Health Records: A Comprehensive Study of Patient Data Extraction and Retrieval
Université Paul Sabatier(保罗·萨巴蒂埃大学);IRIT(信息论与电信研究所);Centre Hospitalier Universitaire de Toulouse(图卢兹大学医院中心);INSERM(法国国家健康与医学研究院);Université de Toulouse(图卢兹大学);Faculté de Médecine(医学院)
QuaLLM-Health: An Adaptation of an LLM-Based Framework for Quantitative Data Extraction from Online Health Discussions
The University of Texas MD Anderson Cancer Center(德克萨斯大学MD安德森癌症中心);Texas Tech Health Science Center El Paso(德克萨斯理工大学埃尔帕索健康科学中心);University of Texas at Austin(德克萨斯大学奥斯汀分校)
K-Bench: a clinically calibrated benchmark for evaluating large language models in high-risk mental health conversations
University of Roehampton(罗汉普顿大学);Kivira Health;University of Hertfordshire(赫特福德大学);University of Surrey(萨里大学);University of Bedfordshire(贝德福德大学);Tavistock Relationships(塔维斯托克关系中心);InsideOut
KokushiMD-10: Benchmark for Evaluating Large Language Models on Ten Japanese National Healthcare Licensing Examinations
Kyoto University(京都大学);The Hong Kong University of Science and Technology(香港科技大学);Xi’an Jiaotong-Liverpool University(西安交通大学利物浦大学);The University of Tokyo(东京大学)
Scalable Medication Extraction and Discontinuation Identification from Electronic Health Records Using Large Language Models
Harvard T.H. Chan School of Public Health, Harvard University(哈佛大学陈曾熙公共卫生学院);Broad Institute of MIT and Harvard(麻省理工学院与哈佛大学博德研究所);Harvard Data Science Initiative, Harvard University(哈佛大学数据科学计划)
Foundation Metrics for Evaluating Effectiveness of Healthcare Conversations Powered by Generative AI
University of California, Irvine(加州大学欧文分校);Stanford University(斯坦福大学);University of Pittsburgh(匹兹堡大学);University of Toronto(多伦多大学);HealthUnity;National Institute of Standards and Technology (NIST)(美国国家标准与技术研究院)
TrustMH-Bench: A Comprehensive Benchmark for Evaluating the Trustworthiness of Large Language Models in Mental Health
Renmin University of China, China(中国人民大学);Beijing University of Posts and Telecommunications, China(北京邮电大学);Hefei University of Technology, China(合肥工业大学)
Still Not Quite There! Evaluating Large Language Models for Comorbid Mental Health Diagnosis
University of Southern California(美国南加州大学);Georgia Institute of Technology(佐治亚理工学院);Sion Hospital(Sion医院);Clinical Psychotherapist(临床心理治疗师);Sophia College for Women(Sophia女子学院);Indian Institute of Technology Bombay(印度班加罗尔理工学院)
Spurious Correlations and Beyond: Understanding and Mitigating Shortcut Learning in SDOH Extraction with Large Language Models
Department of Computer Science(计算机科学系);School of Nursing(护理学院);Department of Information Sciences and Technology(信息科学与技术系);Department of Biomedical Informatics & Medical Education(生物医学信息学与医学教育系)
Extracting Social Determinants of Health from Pediatric Patient Notes Using Large Language Models: Novel Corpus and Methods
University of Washington(华盛顿大学);George Mason University(乔治梅森大学);Dana-Farber Cancer Institute(达纳-法伯癌症研究所);Boston Children's Hospital(波士顿儿童医院);Harvard Medical School(哈佛医学院)
Developing and evaluating a chatbot to support maternal health care
Carnegie Mellon University(卡内基梅隆大学);Population Council Institute(人口理事会研究所);Sitaram Bhartia Institute of Science and Research(西塔拉姆·巴提亚科学与研究研究所);Nivi, Inc.(Nivi公司)
Cognitive-Mental-LLM: Evaluating Reasoning in Large Language Models for Mental Health Prediction via Online Text
Ira A. Fulton Schools of Engineering Arizona State University Tempe, USA(伊拉·A·福林工程学校亚利桑那州立大学Tempe分校);Department of Psychology Santa Clara University Santa Clara, USA(心理学系圣克拉拉大学Santa Clara分校)
Dutch Metaphor Extraction from Cancer Patients' Interviews and Forum Data using LLMs and Human in the Loop
Biomedical Data Sciences, Leiden University Medical Center(莱顿大学医学中心生物医学数据科学);Leiden Institute of Advanced Computer Science (LIACS), Leiden University(莱顿大学先进计算机科学研究所);Department of Radiation Oncology (MAASTRO), GROW Research Institute for Oncology(肿瘤学研究与生殖学研究所放射肿瘤科)
Patterns vs. Patients: Evaluating LLMs against Mental Health Professionals on Personality Disorder Diagnosis through First-Person Narratives
IDEAS Research Institute(IDEAS研究 institute);Adam Mickiewicz University(亚当·密茨凯维奇大学);AMU Center for Artificial Intelligence(AMU人工智能中心);Poznań University of Medical Sciences(波兹南医学科学大学);Maria Curie-Skłodowska University(玛丽·居里-斯克洛多夫斯卡大学)