TCNCA：采用分块注意力的时间卷积网络，用于可扩展序列处理
TCNCA: Temporal Convolution Network with Chunked Attention for Scalable Sequence Processing
AI总结：
本文提出 TCNCA，用可获得大感受野的扩张时间卷积和分块注意力替代 MEGA 的 FFT 并行线性循环，将复杂度降至 $O(L)$，并在语言建模、LRA 和 associative recall 上展现更低损失或同等精度及显著加速。
摘要：
MEGA 是近期提出的一种基于 transformer 的架构，它使用线性循环算子，其基于 FFT 的并行计算复杂度为 $O(LlogL)$，其中 $L$ 为序列长度。我们在其方法基础上，用一种特殊的时间卷积网络替换线性循环；该网络能够以更浅的网络获得更大的感受野，并将计算复杂度降至 $O(L)$。所得模型称为 TCNCA，即带分块注意力的时间卷积网络（Temporal Convolutional Network with Chunked Attention）。我们在 EnWik8 语言建模、long-range-arena（LRA）序列分类以及合成推理基准 associative recall 上评估了 TCNCA。在 EnWik8 上，TCNCA 优于 MEGA，达到更低损失，同时训练期间前向/反向传播速度快 $1.37\ imes$/$1.24\ imes$。TCNCA 使用的扩张卷积在 GPU 上始终显著快于基于 FFT 的并行循环，因此成为处理超长序列的可扩展候选方案：对于长度最高达 131k 的序列，其前向/反向传播速度最高快 $7.07\ imes$/$2.86\ imes$。此外，在 LRA 上，TCNCA 在推理时平均实现 $1.28\ imes$ 加速，且准确率与 MEGA 所达到的水平相近。在 associative recall 上，我们发现，即使是不含过多乘法和加法交互的简化版 TCNCA，在一系列序列长度和词表规模下仍优于 MEGA 或与其具有竞争力。
英文摘要：
MEGA is a recent transformer-based architecture, which utilizes a linear recurrent operator whose parallel computation, based on the FFT, scales as $O(LlogL)$, with $L$ being the sequence length. We build upon their approach by replacing the linear recurrence with a special temporal convolutional network which permits larger receptive field size with shallower networks, and reduces the computational complexity to $O(L)$. The resulting model is called TCNCA, a Temporal Convolutional Network with Chunked Attention. We evaluate TCNCA on EnWik8 language modeling, long-range-arena (LRA) sequence classification, as well as a synthetic reasoning benchmark associative recall. On EnWik8, TCNCA outperforms MEGA, reaching a lower loss with $1.37\times$/$1.24\times$ faster forward/backward pass during training. The dilated convolutions used in TCNCA are consistently and significantly faster operations than the FFT-based parallelized recurrence in GPUs, making them a scalable candidate for handling very large sequence lengths: they are up to $7.07\times$/$2.86\times$ faster in the forward/backward pass for sequences up to 131k. Further on LRA, TCNCA achieves, on average, $1.28\times$ speed-up during inference with similar accuracy to what MEGA achieves. On associative recall, we find that even a simplified version of TCNCA, without excessive multiplicative and additive interactions, remains superior or competitive to MEGA on a range of sequence lengths and vocabulary sizes.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
