arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2408.09158cs.LGcs.AI
收藏

线性注意力足以应对时空预测

Linear Attention is Enough in Spatial-Temporal Forecasting

University of Electronic Science and Technology of China(电子科技大学)

Xinyu Ning

AI总结:

针对交通预测中现有时空图方法难以捕捉动态拓扑、存在过平滑等问题,提出将不同时间步路网节点作为时空令牌输入Transformer的STformer及基于Nyström方法的线性复杂度变体NSTformer,以较低成本实现SOTA性能。

摘要:

作为时空预测任务最具代表性的场景,交通预测任务因其在空间和时间维度上的复杂关联性,受到了机器学习领域的广泛关注。现有方法通常将随时间变化的路网视为时空图,分别处理空间和时间表征。然而,这些方法难以捕捉路网的动态拓扑结构,存在消息传递机制和过平滑问题,且在分别学习时空关系上面临挑战。为解决这些局限,我们提出将不同时间步的路网节点视为独立的时空令牌(token),将其输入普通Transformer以学习复杂的时空模式,设计出达到SOTA(当前最优)的STformer。鉴于其二次复杂度,我们引入基于Nyström方法的变体NSTformer,以线性复杂度近似自注意力,令人惊讶的是,在少数场景下其表现甚至略优于前者。在交通数据集上的大量实验结果表明,所提方法在可承受的计算成本下实现了当前最优性能。我们的代码可在https://github.com/XinyuNing/STformer-and-NSTformer获取。

英文摘要:

As the most representative scenario of spatial-temporal forecasting tasks, the traffic forecasting task attracted numerous attention from machine learning community due to its intricate correlation both in space and time dimension. Existing methods often treat road networks over time as spatial-temporal graphs, addressing spatial and temporal representations independently. However, these approaches struggle to capture the dynamic topology of road networks, encounter issues with message passing mechanisms and over-smoothing, and face challenges in learning spatial and temporal relationships separately. To address these limitations, we propose treating nodes in road networks at different time steps as independent spatial-temporal tokens and feeding them into a vanilla Transformer to learn complex spatial-temporal patterns, design \textbf{STformer} achieving SOTA. Given its quadratic complexity, we introduce a variant \textbf{NSTformer} based on Nystr$\ddot{o}$m method to approximate self-attention with linear complexity but even slightly better than former in a few cases astonishingly. Extensive experimental results on traffic datasets demonstrate that the proposed method achieves state-of-the-art performance at an affordable computational cost. Our code is available at \href{https://github.com/XinyuNing/STformer-and-NSTformer}{https://github.com/XinyuNing/STformer-and-NSTformer}.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑