arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.10448cs.LGcs.AI
收藏

面向轻量级时间序列预测:一种带弱数据增强的补丁式Transformer

Towards Lightweight Time Series Forecasting: a Patch-wise Transformer with Weak Data Enriching

School of Data Science and Engineering, East China Normal University(华东师范大学数据科学与工程学院);School of Computer Science and Technology, Xidian University(西安电子科技大学计算机科学与技术学院);Shanghai Yunxi Technology(上海云玺科技)

Meng Wang, Jintao Yang, Bin Yang, Hui Li, Tongxin Gong, Bo Yang, Jiangtao Cui

AI总结:

针对补丁式Transformer模型复杂、推理昂贵且忽略上下文信息的问题,提出轻量级LiPFormer,采用简化注意力并引入弱数据增强,在九个基准上精度领先且推理时间仅为经典模型的1/3。

摘要:

基于补丁式Transformer的时间序列预测取得了优越的精度。然而,这种优越性严重依赖于具有大量参数的复杂模型设计,使得训练和推理成本高昂,从而阻碍了其在资源有限且低延迟要求的边缘设备上的部署。此外,现有方法通常以自回归方式工作,仅考虑历史值,而忽略了有价值且易于获取的上下文信息,如天气预报、日期和一天中的时间。为应对这两个局限性,我们提出了LiPFormer,一种新颖的带弱数据增强的轻量级补丁式Transformer。首先,为简化Transformer主干,LiPFormer采用了一种新颖的轻量级跨补丁注意力和基于线性变换的注意力,以消除现有Transformer中的两个重组件:层归一化和前馈网络。其次,我们提出了一种轻量级的弱数据增强模块,为训练提供额外的、有价值的弱监督。它提高了预测精度,而不会显著增加模型复杂度,因为它不涉及昂贵的人工标注,而是使用易于获取的上下文信息。这促进了弱数据增强在现有模型上的即插即用。在九个基准时间序列数据集上的大量实验表明,LiPFormer在精度上优于最先进的方法,同时显著减少了参数规模、训练时长和GPU内存使用。在边缘设备上的部署表明,与经典Transformer相比,LiPFormer仅需1/3的推理时间。此外,我们证明了弱数据增强可以无缝集成到各种基于Transformer的模型中以提高其精度,这表明了其通用性。

英文摘要:

Patch-wise Transformer based time series forecasting achieves superior accuracy. However, this superiority relies heavily on intricate model design with massive parameters, rendering both training and inference expensive, thus preventing their deployments on edge devices with limited resources and low latency requirements. In addition, existing methods often work in an autoregressive manner, which take into account only historical values, but ignore valuable, easy-to-obtain context information, such as weather forecasts, date and time of day. To contend with the two limitations, we propose LiPFormer, a novel Lightweight Patch-wise Transformer with weak data enriching. First, to simplify the Transformer backbone, LiPFormer employs a novel lightweight cross-patch attention and a linear transformation-based attention to eliminate Layer Normalization and Feed Forward Network, two heavy components in existing Transformers. Second, we propose a lightweight, weak data enriching module to provide additional, valuable weak supervision to the training. It enhances forecasting accuracy without significantly increasing model complexity as it does not involve expensive, human-labeling but using easily accessible context information. This facilitates the weak data enriching to plug-and-play on existing models. Extensive experiments on nine benchmark time series datasets demonstrate that LiPFormer outperforms state-of-the-art methods in accuracy, while significantly reducing parameter scale, training duration, and GPU memory usage. Deployment on an edge device reveals that LiPFormer takes only 1/3 inference time compared to classic Transformers. In addition, we demonstrate that the weak data enriching can integrate seamlessly into various Transformer based models to enhance their accuracy, suggesting its generality.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑