arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.19802cs.CLcs.AI
收藏

开发医疗保健语言模型嵌入空间

Developing Healthcare Language Model Embedding Spaces

University of Oxford(牛津大学);NHS England(英格兰国家医疗服务体系);University of Liverpool(利物浦大学)

Niall Taylor, Dan Schofield, Andrey Kormilitzin, Dan W Joyce, Alejo Nevado-Holgado

AI总结:

本文探索通过掩码语言建模、DeCLUTR对比学习和元数据预训练三种方法使小型LLMs适应医疗数据集,发现对比训练在文档分类上表现最佳,且领域适应模型均优于通用基础模型,为低计算预算下的医疗LLM专业化部署提供了有效方案。

摘要:

预训练大语言模型(LLMs)在医疗保健文本等分布外数据集上往往表现不佳。我们探索了专门的预训练方法,以使较小的LLMs适应不同的医疗保健数据集。评估了三种方法:传统的掩码语言建模、用于无监督文本表示的深度对比学习(DeCLUTR),以及一种利用医疗保健环境中元数据类别的新型预训练目标。这些方案在每个数据集的下游文档分类任务上进行了评估,并对生成的嵌入空间进行了额外分析。对比训练的模型在分类任务上优于其他方法,在有限的标注数据和较少的模型参数更新下表现出强劲性能。虽然基于元数据的预训练未能进一步提高各数据集的分类效果,但它产生了有趣的嵌入簇可分性。所有领域适应的LLMs都优于其公开可用的通用基础LLM,验证了领域专业化的重要性。这项研究展示了在紧凑型LLMs中灌输医疗保健能力的高效方法,即使在计算预算紧张的情况下也是如此,这是在本地医疗保健环境中负责任和可持续部署的一项基本能力。我们为专门的医疗保健LLMs提供了预训练指南,推动了对对比目标的持续研究,并展示了使小型LLMs与隐私敏感的医疗任务对齐的适应技术。

英文摘要:

Pre-trained Large Language Models (LLMs) often struggle on out-of-domain datasets like healthcare focused text. We explore specialized pre-training to adapt smaller LLMs to different healthcare datasets. Three methods are assessed: traditional masked language modeling, Deep Contrastive Learning for Unsupervised Textual Representations (DeCLUTR), and a novel pre-training objective utilizing metadata categories from the healthcare settings. These schemes are evaluated on downstream document classification tasks for each dataset, with additional analysis of the resultant embedding spaces. Contrastively trained models outperform other approaches on the classification tasks, delivering strong performance from limited labeled data and with fewer model parameter updates required. While metadata-based pre-training does not further improve classifications across the datasets, it yields interesting embedding cluster separability. All domain adapted LLMs outperform their publicly available general base LLM, validating the importance of domain-specialization. This research illustrates efficient approaches to instill healthcare competency in compact LLMs even under tight computational budgets, an essential capability for responsible and sustainable deployment in local healthcare settings. We provide pre-training guidelines for specialized healthcare LLMs, motivate continued inquiry into contrastive objectives, and demonstrates adaptation techniques to align small LLMs with privacy-sensitive medical tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑