arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2112.15087cs.LGcs.AI
收藏

ChunkFormer:基于多级分块的长时间序列学习算法Transformer

ChunkFormer: Learning Long Time Series with Multi-stage Chunked Transformer

Global AI Accelerator, Ericsson(爱立信全球人工智能加速器)

Yue Ju, Alka Isac, Yimin Nie

摘要:

长序列数据的分析在许多实际应用中仍然具有挑战性。我们提出了一种新的架构,ChunkFormer,它改进了现有的Transformer框架,以应对挑战,同时处理长时间序列。原始的基于transformer的模型采用注意力机制来发现沿着序列的全局信息,以利用上下文数据。长序列数据捕获局部信息,例如短数据序列中的季节性和波动。此外,原来的Transformer在训练课程中通过承载整个注意力矩阵消耗更多的资源。为了克服这些挑战,ChunkFormer将长序列拆分为较小的序列块用于注意力计算,并在每个阶段逐步应用不同的块大小。通过这种方式,所提出的模型逐渐学习局部和全局信息,而不改变输入序列的总长度。我们已经在不同的业务领域广泛地测试了这种新架构的有效性,并证明了这种模型相对于现有的基于transformer的模型的优势。

英文摘要:

The analysis of long sequence data remains challenging in many real-world applications. We propose a novel architecture, ChunkFormer, that improves the existing Transformer framework to handle the challenges while dealing with long time series. Original Transformer-based models adopt an attention mechanism to discover global information along a sequence to leverage the contextual data. Long sequential data traps local information such as seasonality and fluctuations in short data sequences. In addition, the original Transformer consumes more resources by carrying the entire attention matrix during the training course. To overcome these challenges, ChunkFormer splits the long sequences into smaller sequence chunks for the attention calculation, progressively applying different chunk sizes in each stage. In this way, the proposed model gradually learns both local and global information without changing the total length of the input sequences. We have extensively tested the effectiveness of this new architecture on different business domains and have proved the advantage of such a model over the existing Transformer-based models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑