arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2510.09776cs.LGcs.AIstat.ML
收藏

为什么 Transformer 无法在上下文中预测时间序列?

Why Do Transformers Fail to Forecast Time Series In-Context?

Duke University(杜克大学);University of Pennsylvania(宾夕法尼亚大学)

Yufa Zhou, Yixiao Wang, Surbhi Goel, Anru R. Zhang

AI总结:

本文从上下文学习理论视角分析了Transformer在时间序列预测中的局限性,证明线性自注意力模型无法超越经典线性模型,且在CoT推理下预测会指数级坍缩到均值,为设计更有效的预测架构提供了理论见解。

摘要:

时间序列预测(TSF)仍然是机器学习中一个具有挑战性且很大程度上未解决的问题,尽管最近在利用大语言模型(LLMs)方面做出了重大努力,而这些模型主要依赖 Transformer 架构。经验证据一致表明,在 TSF 任务上,即使是强大的 Transformer 也往往无法超越简单得多的模型(例如线性模型);然而,对这一现象的严格理论理解仍然有限。在本文中,我们通过上下文学习(ICL)理论的视角对 Transformer 在 TSF 中的局限性进行了理论分析。具体而言,在 AR($p$) 数据下,我们证明了:(1) 线性自注意力(LSA)模型在上下文预测中$\textit{无法}$实现比经典线性模型更低的期望 MSE;(2) 随着上下文长度趋近于无穷大,LSA 渐近地恢复最优线性预测器;(3) 在思维链(CoT)风格的推理下,预测会指数级地坍缩到均值。我们通过精心设计的实验在经验上验证了这些发现。我们的理论不仅阐明了几种以前未被充分探索的现象,还为设计更有效的预测架构提供了实用见解。我们希望我们的工作能鼓励更广泛的研究界重新审视 TSF 的基本理论局限性,并批判性地评估在缺乏更深入审查的情况下直接应用日益复杂的架构的做法。

英文摘要:

Time series forecasting (TSF) remains a challenging and largely unsolved problem in machine learning, despite significant recent efforts leveraging Large Language Models (LLMs), which predominantly rely on Transformer architectures. Empirical evidence consistently shows that even powerful Transformers often fail to outperform much simpler models, e.g., linear models, on TSF tasks; however, a rigorous theoretical understanding of this phenomenon remains limited. In this paper, we provide a theoretical analysis of Transformers' limitations for TSF through the lens of In-Context Learning (ICL) theory. Specifically, under AR($p$) data, we establish that: (1) Linear Self-Attention (LSA) models $\textit{cannot}$ achieve lower expected MSE than classical linear models for in-context forecasting; (2) as the context length approaches to infinity, LSA asymptotically recovers the optimal linear predictor; and (3) under Chain-of-Thought (CoT) style inference, predictions collapse to the mean exponentially. We empirically validate these findings through carefully designed experiments. Our theory not only sheds light on several previously underexplored phenomena but also offers practical insights for designing more effective forecasting architectures. We hope our work encourages the broader research community to revisit the fundamental theoretical limitations of TSF and to critically evaluate the direct application of increasingly sophisticated architectures without deeper scrutiny.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑