arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2402.09558cs.AIcs.LG
收藏

用于改进医疗时间序列表示学习的双向生成式预训练

Bidirectional Generative Pre-training for Improving Healthcare Time-series Representation Learning

School of Computer Science, McGill University（麦吉尔大学计算机学院）；School of Population and Global Health, McGill University（麦吉尔大学人口与全球健康学院）；Mila - Quebec AI Institute（米拉-魁北克人工智能研究所）

Ziyang Song, Qincheng Lu, He Zhu, David Buckeridge, Yue Li

阅读 PDF
arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对医疗时间序列判别任务，提出BiTimelyGPT架构，通过交替Transformer层的双向词元预测进行预训练，保留原始数据分布并提升表示能力，在多项医疗预测任务中表现卓越。

摘要：

在医疗领域，学习用于分类和回归等判别性任务的时间序列表示一直是一个长期存在的挑战。当前的预训练方法仅限于单向的下一词元预测或随机掩码词元预测。我们提出了一种名为双向及时生成式预训练Transformer（BiTimelyGPT）的新架构，该架构通过在交替的Transformer层中进行下一词元和上一词元预测，对生物信号和纵向临床记录进行预训练。此预训练任务保留了时间序列的原始分布和数据形状。此外，满秩的前向和后向注意力矩阵展现出更具表达力的表示能力。使用生物信号和纵向临床记录，BiTimelyGPT在预测神经功能、疾病诊断和生理体征方面表现出卓越的性能。通过可视化注意力热图，我们观察到预训练的BiTimelyGPT能够从生物信号时间序列中识别出判别性片段，且在针对任务进行微调后这种能力更为显著。

英文摘要：

Learning time-series representations for discriminative tasks, such as classification and regression, has been a long-standing challenge in the healthcare domain. Current pre-training methods are limited in either unidirectional next-token prediction or randomly masked token prediction. We propose a novel architecture called Bidirectional Timely Generative Pre-trained Transformer (BiTimelyGPT), which pre-trains on biosignals and longitudinal clinical records by both next-token and previous-token prediction in alternating transformer layers. This pre-training task preserves original distribution and data shapes of the time-series. Additionally, the full-rank forward and backward attention matrices exhibit more expressive representation capabilities. Using biosignals and longitudinal clinical records, BiTimelyGPT demonstrates superior performance in predicting neurological functionality, disease diagnosis, and physiological signs. By visualizing the attention heatmap, we observe that the pre-trained BiTimelyGPT can identify discriminative segments from biosignal time-series sequences, even more so after fine-tuning on the task.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

↑