CEReBrO：使用高效交替注意力的脑振荡表示紧凑编码器
CEReBrO: Compact Encoder for Representations of Brain Oscillations Using Efficient Alternating Attention
AI总结：
本文提出小型EEG基础模型CEReBrO，通过逐通道patch分词与交替注意力联合建模时空信息，以更低内存和更快速度在多项EEG任务上取得领先或竞争性能。
摘要：
脑电图（EEG）是研究大脑活动的关键工具。近来，利用大型无标注数据集的自监督学习方法已成为解决广泛可用的带标注EEG数据稀缺问题的一种潜在方案。然而，当前方法至少存在以下局限之一：i）EEG信号建模次优，ii）模型规模达数亿可训练参数，以及iii）依赖私有数据集和/或不一致的公开基准，阻碍了可复现性。为应对这些挑战，我们提出了用于脑振荡表示的交替注意力紧凑编码器（CEReBrO），这是一种新型小型EEG基础模型。我们的分词方案以逐通道patch粒度表示EEG信号。我们提出一种交替注意力机制，联合建模通道内时间动态与通道间空间相关性，与标准自注意力相比实现2倍速度提升，且所需内存减少6倍。我们提供了从360万到8500万参数的多种模型规模。我们的模型在超过20000小时、具有多样通道配置的公开头皮EEG记录上进行预训练，在情绪检测和癫痫检测任务上树立了新的基准，并在异常分类和步态预测中取得具有竞争力的性能。这验证了我们模型的有效性和效率。
英文摘要：
Electroencephalograph (EEG) is a crucial tool for studying brain activity. Recently, self-supervised learning methods leveraging large unlabeled datasets have emerged as a potential solution to the scarcity of widely available annotated EEG data. However, current methods suffer from at least one of the following limitations: i) sub-optimal EEG signal modeling, ii) model sizes in the hundreds of millions of trainable parameters, and iii) reliance on private datasets and/or inconsistent public benchmarks, hindering reproducibility. To address these challenges, we introduce a Compact Encoder for Representations of Brain Oscillations using alternating attention (CEReBrO), a new small EEG foundation model. Our tokenization scheme represents EEG signals at a per-channel patch granularity. We propose an alternating attention mechanism that jointly models intra-channel temporal dynamics and inter-channel spatial correlations, achieving 2x speed improvement with 6x less memory required compared to standard self-attention. We present several model sizes ranging from 3.6 million to 85 million parameters. Pre-trained on over 20,000 hours of publicly available scalp EEG recordings with diverse channel configurations, our models set new benchmarks in emotion detection and seizure detection tasks, with competitive performance in anomaly classification and gait prediction. This validates our models' effectiveness and efficiency.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
