BlackGoose Rimer:利用RWKV-7作为大规模时间序列建模中Transformer的简单而优越的替代方案
BlackGoose Rimer: Harnessing RWKV-7 as a Simple yet Superior Replacement for Transformers in Large-Scale Time Series Modeling
AI总结:
本文提出BlackGoose Rimer,将RWKV-7的时间混合与通道混合组件集成到Timer模型中,以更少参数实现1.13至43.3倍性能提升和4.5倍训练时间缩减,为大规模时间序列建模提供了优于Transformer的替代方案。
摘要:
时间序列模型在扩展到大型复杂数据集时面临重大挑战,类似于大型语言模型(LLMs)所实现的扩展。时间序列数据的独特特性和模型扩展的计算需求需要创新方法。虽然研究者已经探索了Transformer、LSTM和GRU等多种架构来应对这些挑战,但我们提出了一种使用RWKV-7的新解决方案,该方案将元学习纳入其状态更新机制。通过将RWKV-7的时间混合和通道混合组件集成到基于Transformer的时间序列模型Timer中,我们实现了约1.13至43.3倍的显著性能提升,训练时间减少4.5倍,参数量仅为原来的1/23,同时使用更少的参数。我们的代码和模型权重已公开,供进一步研究和开发使用,地址为https://github.com/Alic-Li/BlackGoose_Rimer。
英文摘要:
Time series models face significant challenges in scaling to handle large and complex datasets, akin to the scaling achieved by large language models (LLMs). The unique characteristics of time series data and the computational demands of model scaling necessitate innovative approaches. While researchers have explored various architectures such as Transformers, LSTMs, and GRUs to address these challenges, we propose a novel solution using RWKV-7, which incorporates meta-learning into its state update mechanism. By integrating RWKV-7's time mix and channel mix components into the transformer-based time series model Timer, we achieve a substantial performance improvement of approximately 1.13 to 43.3x and a 4.5x reduction in training time with 1/23 parameters, all while utilizing fewer parameters. Our code and model weights are publicly available for further research and development at https://github.com/Alic-Li/BlackGoose_Rimer.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。