CAPS:在基于变换器的时间序列预测中统一注意力、递归和对齐
CAPS: Unifying Attention, Recurrence, and Alignment in Transformer-based Time Series Forecasting
AI总结:
CAPS通过结合SO(2)旋转和三种加法门控路径,实现了在时间序列预测中统一注意力、递归和对齐,提升了长期和短期预测性能。
摘要:
本文提出了CAPS(带有前缀乘积和softmax的钟权重聚合),一种用于时间序列预测的结构化注意力机制,该机制解耦了三种不同的时间结构:全球趋势、局部冲击和季节性模式。标准softmax注意力通过全局归一化将这些结构交织在一起,而最近的递归模型为顺序依赖的因果结构牺牲了长期、非顺序依赖的选择。CAPS在单个注意力层内结合了SO(2)旋转用于相位对齐,以及三个加法门控路径——黎曼softmax、前缀乘积门和钟基线——通过共享的时间重要性概念调节这些路径。实验结果在长期和短期预测基准上超过了 vanilla softmax 和线性注意力机制,并在与七个强大的基线模型竞争时展示了具有竞争力的性能。我们的代码实现可在https://github.com/vireshpati/CAPS-Attention上获得。
英文摘要:
This paper presents $\textbf{CAPS}$ (Clock-weighted Aggregation with Prefix-products and Softmax), a structured attention mechanism for time series forecasting that decouples three distinct temporal structures: global trends, local shocks, and seasonal patterns. Standard softmax attention entangles these through global normalization, while recent recurrent models sacrifice long-term, order-independent selection for order-dependent causal structure. CAPS combines SO(2) rotations for phase alignment with three additive gating paths -- Riemann softmax, prefix-product gates, and a Clock baseline -- within a single attention layer. We introduce the Clock mechanism, a learned temporal weighting that modulates these paths through a shared notion of temporal importance. Experiments on long- and short-term forecasting benchmarks surpass vanilla softmax and linear attention mechanisms and demonstrate competitive performance against seven strong baselines with linear complexity. Our code implementation is available at https://github.com/vireshpati/CAPS-Attention.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。