注意力机制的诅咒：基于核方法的视角探讨Transformer在时间序列预测及更多任务上泛化失败的原因
Curse of Attention: A Kernel-Based Perspective for Why Transformers Fail to Generalize on Time Series Forecasting and Beyond
AI总结：
本文从核方法视角解释了Transformer在时间序列预测任务中泛化失败的原因，指出注意力机制在符号不一致时存在非对称学习问题，导致无法学习残差特征，为设计高效Transformer架构提供了理论依据。
摘要：
基于Transformer的模型在时间序列预测（TSF）任务中的应用长期以来备受研究关注。然而，许多此类工作无法超越简单的线性残差模型，且对这一问题的理论理解仍然有限。在本研究中，我们首次提出了关于Transformer在TSF任务上低效问题的理论解释。我们将其背后的机制归结为训练注意力网络时的“非对称学习”。在下一步预测的时间序列中，当上一步的符号与当前步的符号不一致时，注意力机制无法学习到残差特征。这导致模型难以在分布外（OOD）数据上泛化，尤其是对于具有相同表示模式但符号不一致的下一步预测数据，而线性残差网络却能轻松完成这一任务。我们希望这些理论见解能为从业者设计具有表达力且高效的基于Transformer的架构提供重要的必要条件。
英文摘要：
The application of transformer-based models on time series forecasting (TSF) tasks has long been popular to study. However, many of these works fail to beat the simple linear residual model, and the theoretical understanding of this issue is still limited. In this work, we propose the first theoretical explanation of the inefficiency of transformers on TSF tasks. We attribute the mechanism behind it to {\bf Asymmetric Learning} in training attention networks. When the sign of the previous step is inconsistent with the sign of the current step in the next-step-prediction time series, attention fails to learn the residual features. This makes it difficult to generalize on out-of-distribution (OOD) data, especially on the sign-inconsistent next-step-prediction data, with the same representation pattern, whereas a linear residual network could easily accomplish it. We hope our theoretical insights provide important necessary conditions for designing the expressive and efficient transformer-based architecture for practitioners.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
