arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2105.06752cs.CL
收藏

使用预训练的Transformers分类长临床文档

Classifying Long Clinical Documents with Pre-trained Transformers

University of Arizona(亚利桑那大学);Boston Children’s Hospital(波士顿儿童医院);Harvard Medical School(哈佛医学院);University of Wisconsin–Madison(威斯康星大学麦迪逊分校);Loyola University Chicago(芝加哥洛约拉大学)

Xin Su, Timothy Miller, Xiyu Ding, Majid Afshar, Dmitriy Dligach

摘要:

自动表型分析是识别与预定义的一组标准匹配的患者队列的任务。表型分析通常涉及对包含数千个标记的长临床文档进行分类。同时,最近最先进的基于transformer的预训练语言模型将输入限制为几百个标记(例如,对于BERT为512个标记)。我们评估了将预训练的句子编码器纳入临床文本的文档级表示的几种策略,并发现没有预训练的分层Transformers与任务预训练模型具有竞争力。

英文摘要:

Automatic phenotyping is a task of identifying cohorts of patients that match a predefined set of criteria. Phenotyping typically involves classifying long clinical documents that contain thousands of tokens. At the same time, recent state-of-art transformer-based pre-trained language models limit the input to a few hundred tokens (e.g. 512 tokens for BERT). We evaluate several strategies for incorporating pre-trained sentence encoders into document-level representations of clinical text, and find that hierarchical transformers without pre-training are competitive with task pre-trained models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑