arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2606.03250cs.CL
收藏

词与道:德语医学NLP中领域特定BERT预训练的策略

The Word and the Way: Strategies for Domain-Specific BERT Pre-Training in German Medical NLP

School of Computation, Information and Technology(计算信息科技学院);Technical University of Munich(慕尼黑技术大学);Chair of IT Infrastructure for Translational Medical Research(转化医学研究信息基础设施主任);Faculty of Applied Computer Science(应用计算机科学学院);University of Augsburg(艾希施泰特大学);Institute of General Practice(基础医学研究所);Faculty of Medicine and Medical Center(医学学院和医学中心)

Henry He, Johann Frei, Raphael Schmitt

AI总结:

本文提出ChristBERT系列模型,通过对比持续预训练、从头训练和领域词汇适应三种策略,在德语医学NLP任务中实现最优性能,并建立新的基准。

摘要:

数字医疗产生大量临床文本,可支持AI辅助应用,但德语生物医学语言模型仍受限于较旧的架构或受限的训练数据。我们提出了ChristBERT(临床与健康相关议题及主题调优BERT),这是一个基于德语RoBERTa的领域特定语言模型家族,在包含科学出版物、临床文本、健康相关网络内容和翻译临床资源的13.5GB语料库上训练。为了探究领域适应策略在德语临床NLP中的影响,我们比较了持续预训练、从头训练和领域词汇适应。所得模型在三个医学命名实体识别任务和两个文本分类任务上进行了评估。ChristBERT在五个基准中的四个上持续优于现有的通用和医学德语语言模型,并为德语临床语言建模建立了新的最先进水平。我们的结果表明,最优适应策略取决于任务:在我们的评估中,从头训练对高度专业化的临床文本特别有效,而持续预训练在更常见的医学文本上表现良好。所有模型均已公开发布,以支持德语医学NLP的未来研究和应用。

英文摘要:

Digital healthcare generates vast amounts of clinical text that can support AI-assisted applications, yet German biomedical language models remain limited by older architectures or restricted training data. We present ChristBERT (Clinical- and Healthcare-Related Issues and Subjects Tuned BERT), a family of domain-specific German RoBERTa-based language models trained on a 13.5GB corpus of scientific publications, clinical texts, health-related web content, and translated clinical resources. To investigate the impact of domain adaptation strategies in German clinical NLP, we compare continued pre-training, training from scratch, and domain-specific vocabulary adaptation. The resulting models are evaluated on three medical named entity recognition tasks and two text classification tasks. ChristBERT consistently outperforms existing general-purpose and medical German language models on four of five benchmarks and establishes a new state of the art for German clinical language modeling. Our results show that the optimal adaptation strategy is task-dependent: in our evaluation, training from scratch is particularly effective for highly specialized clinical texts, whereas continued pre-training performs well on more commonly written medical texts. All models are publicly released to support future research and applications in German medical NLP.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑