用于长序列建模的重参数化多分辨率卷积
Reparameterized Multi-Resolution Convolutions for Long Sequence Modelling
AI总结：
针对长卷积训练难、易过拟合的问题，提出融合多分辨率卷积、结构重参数化与可学习核衰减的$\texttt{MRConv}$，在多任务上取得SOTA并提升ImageNet分类性能。
摘要：
全局卷积作为功能强大的通用序列模型，前景日益广阔。然而，训练长卷积颇具挑战，且核参数化方法必须能够在不过拟合的前提下学习长程依赖关系。本研究提出重参数化多分辨率卷积（$\texttt{MRConv}$），这是一种为长序列建模对全局卷积核进行参数化的全新方法。通过利用多分辨率卷积、引入结构重参数化以及加入可学习的核衰减机制，$\texttt{MRConv}$ 能够学习到表达力强的长程卷积核，在多种数据模态下均表现出色。我们的实验表明，在卷积模型和线性时间 Transformer 中，$\texttt{MRConv}$ 在 Long Range Arena、Sequential CIFAR 以及 Speech Commands 任务上取得了最优性能。此外，我们通过用 1D $\texttt{MRConv}$ 层替换 2D 卷积，在 ImageNet 分类任务上实现了性能提升。
英文摘要：
Global convolutions have shown increasing promise as powerful general-purpose sequence models. However, training long convolutions is challenging, and kernel parameterizations must be able to learn long-range dependencies without overfitting. This work introduces reparameterized multi-resolution convolutions ($\texttt{MRConv}$), a novel approach to parameterizing global convolutional kernels for long-sequence modelling. By leveraging multi-resolution convolutions, incorporating structural reparameterization and introducing learnable kernel decay, $\texttt{MRConv}$ learns expressive long-range kernels that perform well across various data modalities. Our experiments demonstrate state-of-the-art performance on the Long Range Arena, Sequential CIFAR, and Speech Commands tasks among convolution models and linear-time transformers. Moreover, we report improved performance on ImageNet classification by replacing 2D convolutions with 1D $\texttt{MRConv}$ layers.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
