arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.14766cs.CLcs.LG
收藏

评估大型语言模型在公共卫生分类与抽取任务中的表现

Evaluating Large Language Models for Public Health Classification and Extraction Tasks

UK Health Security Agency(英国卫生安全局)
Joshua Harris, Timothy Laurence, Leo Loman, Fan Grayson, Toby Nonnenmacher, Harry Long, Loes WalsGriffith, Amy Douglas, Holly Fountain, Stelios Georgiou, Jo Har… 展开作者

Joshua Harris, Timothy Laurence, Leo Loman, Fan Grayson, Toby Nonnenmacher, Harry Long, Loes WalsGriffith, Amy Douglas, Holly Fountain, Stelios Georgiou, Jo Hardstaff, Kathryn Hopkins, Y-Ling Chi, Galena Kuyumdzhieva, Lesley Larkin, Samuel Collins, Hamish Mohammed, Thomas Finnie, Luke Hounsome, Michael Borowitz, Steven Riley

AI总结:

该研究结合13个标注数据集,以零样本上下文学习评估11个开放权重LLM及GPT-4系列在公共卫生文本分类与抽取任务中的表现,发现Llama-3.3-70B-Instruct整体最优且具备应用潜力。

摘要:

大型语言模型(LLM)的进展引发了人们对其在包括公共卫生在内的多个领域支持人类专家潜力的浓厚兴趣。在本研究中,我们针对涉及自由文本分类与抽取的公共卫生任务,提出了对LLM的自动化评估。我们将6个外部标注数据集与7个新的内部标注数据集相结合,以评估LLM处理与以下内容相关文本的能力:健康负担、流行病学风险因素以及公共卫生干预措施。我们使用零样本上下文学习,在所有任务上评估了11个开放权重LLM(70亿至1230亿参数)。我们发现Llama-3.3-70B-Instruct是表现最好的模型,在16项任务中的8项上取得了最佳结果(使用micro-F1分数)。我们观察到不同任务之间存在显著差异:所有开放权重LLM在某些具有挑战性的任务(如Contact Classification)上的micro-F1得分均低于60%,而所有LLM在其他任务(如GI Illness Classification)上的micro-F1得分均高于80%。对于11项任务的一个子集,我们还评估了三个GPT-4和GPT-4o系列模型,并发现其结果与Llama-3.3-70B-Instruct相当。总体而言,基于这些初步结果,我们看到了令人鼓舞的迹象:LLM可能成为公共卫生专家从各种自由文本来源抽取信息的有用工具,并支持公共卫生监测、研究和干预。

英文摘要:

Advances in Large Language Models (LLMs) have led to significant interest in their potential to support human experts across a range of domains, including public health. In this work we present automated evaluations of LLMs for public health tasks involving the classification and extraction of free text. We combine six externally annotated datasets with seven new internally annotated datasets to evaluate LLMs for processing text related to: health burden, epidemiological risk factors, and public health interventions. We evaluate eleven open-weight LLMs (7-123 billion parameters) across all tasks using zero-shot in-context learning. We find that Llama-3.3-70B-Instruct is the highest performing model, achieving the best results on 8/16 tasks (using micro-F1 scores). We see significant variation across tasks with all open-weight LLMs scoring below 60% micro-F1 on some challenging tasks, such as Contact Classification, while all LLMs achieve greater than 80% micro-F1 on others, such as GI Illness Classification. For a subset of 11 tasks, we also evaluate three GPT-4 and GPT-4o series models and find comparable results to Llama-3.3-70B-Instruct. Overall, based on these initial results we find promising signs that LLMs may be useful tools for public health experts to extract information from a wide variety of free text sources, and support public health surveillance, research, and interventions.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑