发表机构
ETH Zurich; Nanyang Technological University; University of Bologna（苏黎世联邦理工学院; 南洋理工大学; 博洛尼亚大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
该研究针对连续EEG监测中Transformer架构的内存瓶颈，提出S-CEReBrO架构，通过窗口交替注意力实现内存恒定，在EEG下游任务中性能领先且效率提升。
AI 中文摘要
基础模型为脑电图（EEG）分析提供了极具前景的范式，它利用来自大量未标记数据集的可泛化表示。然而，基于Transformer的架构面临一个关键瓶颈：全局注意力机制将注意力内存状态与信号时长绑定，导致连续监测时出现内存溢出。为解决该问题，我们提出S-CEReBrO（Streaming CEReBrO），这是专为连续监测设计的CEReBrO架构的改进版本。我们的新型窗口交替注意力机制将注意力计算分解为固定大小的时空窗口，由于仅活跃窗口需要驻留注意力图，因此能保证KV缓存内存恒定。经验缩放分析证实，窗口交替注意力可处理比全自注意力长100倍、比低秩线性注意力长3倍的信号。在长上下文场景下，与低秩线性注意力相比，窗口交替注意力仅需55%的内存，同时推理吞吐量提升2.1倍。S-CEReBrO在超过12000名受试者的25000多小时记录上进行预训练，在11项下游任务中的7项达到了最先进性能，参数数量最多减少60%。这项工作为实现高效、可泛化的连续EEG监测迈出了重要一步，配套代码仓库已开放。
英文摘要
Foundation models offer a promising paradigm for Electroencephalography (EEG) analysis, leveraging generalizable representations from vast unlabeled datasets. Yet, Transformer-based architectures face a critical bottleneck: global attention mechanisms couple the attention memory state to the signal duration, causing memory overflow during continuous monitoring. To address this, we introduce S-CEReBrO (Streaming CEReBrO), an evolution of the CEReBrO architecture designed for continuous monitoring. Our novel Windowed Alternating Attention mechanism factorizes attention computation into fixed-size spatiotemporal windows, so that under streaming, only the active window remains resident and the attention state is bounded independently of signal duration. Empirical scaling analysis shows that windowed alternating attention can process signals 100X longer than full self-attention and 3X longer than low-rank linear attention. Compared to low-rank linear attention on long contexts, windowed alternating attention requires 55% of the memory while increasing inference throughput by 2.1X. Pre-trained on >25,000 hours of recordings from >12,000 subjects, S-CEReBrO achieves state-of-the-art performance on 7 of 11 downstream tasks, with up to 60% fewer parameters. This work represents a significant step toward the realization of efficient, generalizable, and continuous EEG monitoring. An accompanying code repository is available. An accompanying code repository is available.
CommentsThis is the pre-rebuttal version of a paper accepted at MICCAI 2026. The camera-ready version will be posted following the embargo
