自预训练（SPT）真的有助于改进医疗时间序列的诊断吗？
Is Self-Pretraining really useful to improve diagnosis in medical Time Series?
浏览论文内容
中文总结 AI 辅助
该研究探究自预训练（SPT）对Transformer在医疗时间序列诊断任务的影响，发现SPT可提升分类准确率0-6个百分点，是无需修改架构的通用有效策略。
中文摘要 AI 辅助
受近期研究证据启发——Transformer架构在长基准测试中受益于自预训练（Self-PreTraining, SPT），本文探究类似增益是否可扩展至多模态、多变量乃至简单单变量的医疗时间序列场景。研究目标是评估SPT对Transformer模型在各类医疗应用中性能与可扩展性的影响，尤其是在数据有限的条件下。本文在三类代表性医疗时间序列任务上评估Transformer架构：康复机器人任务（Camargo数据集）、压力检测任务（Non-EEG Stress）、帕金森病检测任务（Gait Parkinson's Disease）。模型采用两种训练方式：从零开始训练，或通过SPT训练，所用的四个基于掩码的目标函数旨在促进时间与跨模态表示学习；同时系统调整模型深度，以探究模型容量与预训练增益的相互作用。在所有数据集与配置下，SPT均能将分类准确率提升0至6个百分点，具体提升幅度取决于掩码策略、数据集与架构；增益不仅出现在多变量场景，当模型仅使用简单单变量输入时也存在。对于更深的模型，其能更好地利用预训练学到的丰富时间表示，提升幅度也更大。这些发现表明，SPT是一种简单通用的策略，无需针对任务修改架构即可提升Transformer在医疗时间序列任务上的性能，支持其在数据有限的临床环境中提升鲁棒性与准确率的潜力。
英文摘要
Inspired by recent evidence that transformer architectures benefit from Self-PreTraining (SPT) on long-context benchmarks, we investigate whether similar gains extend to multimodal, multivariate, and even simple univariate medical time series. Our objective is to assess the impact of SPT on the performance and scalability of transformer-based models across diverse medical applications, particularly under limited data conditions. We evaluate transformer architectures on three representative medical time-series tasks: rehabilitation robotics (Camargo dataset), stress detection (Non-EEG Stress), and Parkinson's disease detection (Gait Parkinson's Disease). Models are trained either from scratch or through SPT using four masking-based objectives designed to promote temporal and cross-modal representation learning, and we systematically vary model depth to examine how capacity interacts with pre-training benefits. Across datasets and configurations, SPT consistently improves classification accuracy by 0-6 percentage points depending on masking strategy, dataset and architecture, with gains observed not only in multivariate settings but also when models are restricted to simple univariate inputs. The improvements increase for deeper models that can better exploit the enriched temporal representations learned during pre-training. These findings indicate that SPT is a simple and general strategy that enhances transformer performance on medical time-series tasks without requiring task-specific architectural changes, supporting its potential to improve robustness and accuracy in data-limited clinical settings.
发表机构
- Università Campus Bio-Medico di Roma（罗马生物医学大学校园大学）
- Umeå University（于默奥大学）
- Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
- ELLIS Institute Tübingen（埃利斯研究所蒂宾根分所）
机构由 AI 辅助整理，请以论文原文为准。
