自适应各向异性注意力用于轴结构化信号
Adaptive Anisotropic Attention for Axis-Structured Signals
浏览论文内容
中文总结 AI 辅助
提出自适应各向异性注意力（AAA），将注意力分解为时间与空间路径，通过门控加权组合，在EEG等轴结构化信号上提升性能，验证了轴对齐归纳偏置的有效性。
中文摘要 AI 辅助
密集自注意力在学习前将所有词元对视为同等可能，这是一种交互各向同性的先验，可能与结构化信号不匹配。对于诸如脑电图（EEG）这类结构化、低信噪比（SNR）的信号，依赖关系沿着电极轴和时间轴组织，而这种均匀先验使每个词元暴露于许多不相关的交互中。我们提出了自适应各向异性注意力（AAA），它将注意力分为两条路径：一条时间路径，其中每个词元在时间维度上关注其自身电极的词元；以及一条空间路径，其中每个词元在同一时间步关注其他电极的词元。一个小的门控为每个词元预测两条路径输出的凸组合：两个非负权重之和为1。在六个EEG下游任务上，所得到的模型AXON（轴因子化算子网络）在线性探测和完全微调下，相较于密集基线均提高了平均平衡准确率。我们证明两条路径（时间路径和空间路径）都是必要的，且加权和优于对单一路径的硬选择；大部分收益来自于门控在网络的每一层学习不同的时间/空间平衡。受控的音频频谱图实验表明，轴因子化可迁移到EEG之外。这些结果表明，将注意力与结构化信号的自然轴对齐提供了一种有用的归纳偏置。
英文摘要
Dense self-attention treats all token pairs as equally plausible before learning, an interaction-isotropic prior that can be mismatched to structured signals. For structured, low signal-to-noise ratio (SNR) signals such as EEG, dependencies are organized along the electrode and time axes, and this uniform prior exposes each token to many irrelevant interactions. We introduce Adaptive Anisotropic Attention (AAA), which splits attention into two paths: a temporal path, where each token attends to the tokens of its own electrode across time, and a spatial path, where it attends to the tokens of the other electrodes at the same time step. A small gate predicts, for every token, a convex combination of the two path outputs: two non-negative weights that sum to one. On six EEG downstream tasks, the resulting model, AXON (AXis-factorized Operator Network), improves mean balanced accuracy over a dense baseline under both linear probing and full fine-tuning. We show that both paths (temporal and spatial) are necessary and that the weighted sum beats a hard choice of one path; most of the benefit comes from the gate learning a different temporal/spatial balance at each layer of the network. Controlled audio spectrogram experiments show that axis factorization transfers beyond EEG. These results suggest that aligning attention with the natural axes of structured signals provides a useful inductive bias.
发表机构
- Mannas AI（曼纳斯人工智能）
机构由 AI 辅助整理，请以论文原文为准。
