站在巨人肩膀上：通过多教师蒸馏重新思考EEG基础模型预训练
Standing on the Shoulders of Giants: Rethinking EEG Foundation Model Pretraining via Multi-Teacher Distillation
AI总结：
本文提出MTDP框架，通过多教师蒸馏预训练EEG基础模型，提升模型性能并减少预训练数据需求。
摘要：
EEG基础模型的预训练主要依赖于自监督的掩码重建，这一范式主要借鉴并受图像和语言基础模型成功的影响。然而，与图像和文本不同，EEG数据集收集成本高昂且信噪比较低。这些挑战在扩大EEG基础模型规模和通过重建捕捉底层神经语义时引入了困难。在本工作中，我们提出问题：能否站在已建立的、在代表性模态中表现良好的基础模型上，来引导EEG基础模型的预训练？我们首先证明主流基础模型，如视觉和时间序列模型，能够惊人地转移到EEG领域。为此，我们提出了多教师蒸馏预训练（MTDP）框架，用于通过两阶段多教师蒸馏预训练EEG基础模型。第一阶段中，我们引入一个可学习的门控网络，通过掩码潜在去噪目标融合来自不同教师（例如DINOv3和Chronos）的表示。第二阶段中，我们将融合的表示蒸馏到EEG基础模型中。在9个下游任务和12个数据集上的广泛评估表明，我们的基于MTDP的EEG基础模型在仅使用25%预训练数据的情况下，优于其自监督的对应物。
英文摘要：
Pretraining for electroencephalogram (EEG) foundation models has predominantly relied on self-supervised masked reconstruction, a paradigm largely adapted from and inspired by the success of vision and language foundation models. However, unlike images and text, EEG datasets are notoriously expensive to collect and characterized by low signal-to-noise ratio. These challenges introduce difficulties in scaling the EEG foundation models and capturing the underlying neural semantics through reconstruction. In this work, we ask the question: can we stand on the shoulders of well-established foundation models from well-represented modalities to bootstrap the pretraining of EEG foundation models? We first demonstrate that mainstream foundation models, such as those from vision and time series, transfer surprisingly well to EEG domain. To this end, we propose the Multi-Teacher Distillation Pretraining (MTDP) framework for pretraining EEG foundation models via a two-stage multi-teacher distillation. In the first stage, we introduce a learnable gating network to fuse representations from diverse teachers (e.g., DINOv3 and Chronos) via a masked latent denoising objective. In the second stage, we distill the fused representation into an EEG foundation model. Extensive evaluations across 9 downstream tasks and 12 datasets demonstrate that our MTDP-based EEG foundation model outperforms its self-supervised counterparts while requiring only 25% of the pretraining data.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
