arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2404.10337cs.AI
收藏

理解基于Transformer的时间序列预测中的Token级拓扑结构

Understanding Token-level Topological Structures in Transformer-based Time Series Forecasting

University of Chinese Academy of Sciences(中国科学院大学);National Key Laboratory of Space Integrated Information System, Institute of Software, Chinese Academy of Sciences(空间信息集成系统国家重点实验室,软件研究所,中国科学院);Wangxuan Institute of Computer Technology, Peking University(王轩计算机技术研究所,北京大学);Hong Kong University of Science and Technology(香港科技大学)

Jianqi Zhang, Wenwen Qiang, Jingyao Wang, Jiahuan Zhou, Changwen Zheng, Hui Xiong

AI总结:

针对Transformer时间序列预测中token拓扑结构随网络加深而退化的问题,提出TEM方法,通过位置与语义拓扑增强模块及双层优化显式保留拓扑,提升预测性能。

摘要:

基于Transformer的方法通过捕获输入token之间的位置和语义拓扑关系,在时间序列预测(TSF)中取得了最先进的性能。然而,现有Transformer是否在中间层充分利用了token间固有的拓扑结构,目前尚不清楚。通过实证和理论分析,我们发现当前的Transformer架构随着网络加深会逐渐退化输入token的原始位置和语义拓扑,从而限制了预测精度。此外,我们的理论结果表明,在中间层显式强制保留这些拓扑结构可以收紧泛化界限,从而带来改进的预测性能。受这些见解的启发,我们提出了拓扑增强方法(TEM),一种新颖的基于Transformer的TSF方法,它显式且自适应地保留token级拓扑。TEM由两个核心模块组成:1)位置拓扑增强模块(PTEM),注入可学习的位置约束以显式保留原始位置拓扑;2)语义拓扑增强模块(STEM),引入可学习的相似性矩阵以保留原始语义拓扑。为了自适应地确定最优注入权重,TEM采用了双层优化策略。所提出的TEM是一种即插即用方法,可以与现有的基于Transformer的TSF方法集成。大量实验表明,将TEM与多种现有方法集成可显著提高其预测性能,验证了显式保留原始token级拓扑的有效性。我们的代码公开于:https://github.com/jlu-phyComputer/TEM。

英文摘要:

Transformer-based methods have achieved state-of-the-art performance in time series forecasting (TSF) by capturing positional and semantic topological relationships among input tokens. However, it remains unclear whether existing Transformers fully leverage the intrinsic topological structure among tokens throughout intermediate layers. Through empirical and theoretical analyses, we identify that current Transformer architectures progressively degrade the original positional and semantic topology of input tokens as the network deepens, thus limiting forecasting accuracy. Furthermore, our theoretical results demonstrate that explicitly enforcing preservation of these topological structures within intermediate layers can tighten generalization bounds, leading to improved forecasting performance. Motivated by these insights, we propose the Topology Enhancement Method (TEM), a novel Transformer-based TSF method that explicitly and adaptively preserves token-level topology. TEM consists of two core modules: 1) the Positional Topology Enhancement Module (PTEM), which injects learnable positional constraints to explicitly retain original positional topology; 2) the Semantic Topology Enhancement Module (STEM), which incorporates a learnable similarity matrix to preserve original semantic topology. To determine optimal injection weights adaptively, TEM employs a bi-level optimization strategy. The proposed TEM is a plug-and-play method that can be integrated with existing Transformer-based TSF methods. Extensive experiments demonstrate that integrating TEM with a variety of existing methods significantly improves their predictive performance, validating the effectiveness of explicitly preserving original token-level topology. Our code is publicly available at: \href{https://github.com/jlu-phyComputer/TEM}{https://github.com/jlu-phyComputer/TEM}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑