Advances in time-series forecasting are driving a shift from conventional machine learning models to foundation models (FMs) that are trained with generalized knowledge. However, existing FMs still perform poorly in the energy fields, such as building energy forecasting (BEF). This paper studies the adaptation of FM to BEF tasks. We demonstrate the shortcomings of fine-tuning FM straightforwardly from both the perspectives of FM and the data. To overcome these limitations, we propose a new \textit{contrastive curriculum learning}-based training method. Our method optimizes the ordering of training data in the context of TSFM adaptation. Experiments show that our method can improve the zero/few-shot performance by 14.6\% compared to the existing FMs. Our code and new TSFM will be available at <Anonymous Github Repo>.
Enhancing Spatio-temporal Quantile Forecasting with Curriculum Learning: Lessons Learned
University of New South Wales（新南威尔士大学）；HKGAI（香港人工智能学会）；Southern University of Science and Technology（南方科技大学）；The University of Tokyo（东京大学）；School of Artificial Intelligence of Jilin University（吉林大学人工智能学院）
For any financial organization, computing accurate quarterly forecasts for various products is one of the most critical operations. As the granularity at which forecasts are needed increases, traditional statistical time series models may not scale well. We apply deep neural networks in the forecasting domain by experimenting with techniques from Natural Language Processing (Encoder-Decoder LSTMs) and Computer Vision (Dilated CNNs), as well as incorporating transfer learning. A novel contribution of this paper is the application of curriculum learning to neural network models built for time se…
University of Amsterdam（阿姆斯特丹大学）；Ahold Delhaize（阿霍德德尔海兹）
In various business settings, there is an interest in using more complex machine learning techniques for sales forecasting. It is difficult to convince analysts, along with their superiors, to adopt these techniques since the models are considered to be "black boxes," even if they perform better than current models in use. We examine the impact of contrastive explanations about large errors on users' attitudes towards a "black-box'" model. We propose an algorithm, Monte Carlo Bounds for Reasonable Predictions. Given a large error, MC-BRP determines (1) feature values that would result in a rea…
Technische Universität Ilmenau（伊尔默瑙工业大学）；Friedrich-Schiller-Universität Jena（耶拿弗里德里希·席勒大学）
Sequence-to-sequence models based on LSTM and GRU are a most popular choice for forecasting time series data reaching state-of-the-art performance. Training such models can be delicate though. The two most common training strategies within this context are teacher forcing (TF) and free running (FR). TF can be used to help the model to converge faster but may provoke an exposure bias issue due to a discrepancy between training and inference phase. FR helps to avoid this but does not necessarily lead to better results, since it tends to make the training slow and unstable instead. Scheduled samp…
Dual-Criterion Curriculum Learning: Application to Temporal Data
Université Paris Saclay, Université Paris Cité, ENS Paris Saclay, CNRS, SSA, INSERM, Centre Borelli（巴黎萨克雷大学、巴黎cité大学、巴黎萨克雷高等师范学院、国家科学研究中心、SSA、国家医学研究院、Borelli中心）；Centre d’Analyse et de Mathématique Sociales, EHESS, CNRS（分析与数学社会研究中心、高等社会科学学院、国家科学研究中心）；EDF R&D（电力公司研发）
RED-F: Reconstruction-Elimination based Dual-stream Contrastive Forecasting for Multivariate Time Series Anomaly Prediction
School of Computer Science (National Pilot Software Engineering School), Beijing University of Posts and Telecommunications（计算机学院（国家级试点软件工程学院），北京邮电大学）；China Telecom Research Institute Beijing（中国电信研究院北京分院）；Department of Computer Science, Missouri University of Science and Technology（计算机学院，密苏里科技大学）
KTH Royal Institute of Technology（瑞典皇家理工学院）；RISE Research Institutes of Sweden（瑞典RISE研究院）
We present a representation learning framework for financial time series forecasting. One challenge of using deep learning models for finance forecasting is the shortage of available training data when using small datasets. Direct trend classification using deep neural networks trained on small datasets is susceptible to the overfitting problem. In this paper, we propose to first learn compact representations from time series data, then use the learned representations to train a simpler model for predicting time series movements. We consider a class-conditioned latent variable model. We train …
Mitigating Data Redundancy to Revitalize Transformer-based Long-Term Time Series Forecasting System
Stanford University（斯坦福大学）；University of Technology Sydney（悉尼科技大学）；CSIRO’s Data61（澳大利亚联邦科学与工业研究组织Data61）；University of New South Wales（新南威尔士大学）；University of Science and Technology of China（中国科学技术大学）
TLCCSP: A Scalable Framework for Enhancing Time Series Forecasting with Time-Lagged Cross-Correlations
School of Artificial Intelligence（人工智能学院）；Institute of Artificial Intelligence and Future Networks（人工智能与未来网络研究所）；Department of Systems Science, Faculty of Arts and Sciences（艺术与科学学院系统科学系）；Guangdong Key Lab of AI and Multi-Modal Data Processing（广东人工智能与多模态数据处理重点实验室）；BNU-HKBU United International College（北师大-香港大学联合国际学院）
In this work, we present LaserFlow, an efficient method for 3D object detection and motion forecasting from LiDAR. Unlike the previous work, our approach utilizes the native range view representation of the LiDAR, which enables our method to operate at the full range of the sensor in real-time without voxelization or compression of the data. We propose a new multi-sweep fusion architecture, which extracts and merges temporal features directly from the range images. Furthermore, we propose a novel technique for learning a probability distribution over future trajectories inspired by curriculum …
James Cook University（詹姆斯库克大学）；College of Science and Engineering（科学与工程学院）
Spatio-temporal sequence forecasting is one of the fundamental tasks in spatio-temporal data mining. It facilitates many real world applications such as precipitation nowcasting, citywide crowd flow prediction and air pollution forecasting. Recently, a few Seq2Seq based approaches have been proposed, but one of the drawbacks of Seq2Seq models is that, small errors can accumulate quickly along the generated sequence at the inference stage due to the different distributions of training and inference phase. That is because Seq2Seq models minimise single step errors only during training, however t…
Beyond Fixed Variables: Expanding-variate Time Series Forecasting via Flat Scheme and Spatio-temporal Focal Learning
Southwest Jiaotong University（西南交通大学）；College of Computer Science and Technology（计算机科学与技术学院）；Zhejiang University（浙江大学）；Engineering Research Center of Sustainable Urban Intelligent Transportation（可持续城市智能交通工程研究中心）；Department of Computer Science（计算机科学系）；Aalborg University（奥胡斯大学）
We introduce EfficientCL, a memory-efficient continual pretraining method that applies contrastive learning with novel data augmentation and curriculum learning. For data augmentation, we stack two types of operation sequentially: cutoff and PCA jittering. While pretraining steps proceed, we apply curriculum learning by incrementing the augmentation degree for each difficulty step. After data augmentation is finished, contrastive learning is applied on projected embeddings of original and augmented examples. When finetuned on GLUE benchmark, our model outperforms baseline models, especially fo…
Time series forecasting is a significant problem in many applications, e.g., financial predictions and business optimization. Modern datasets can have multiple correlated time series, which are often generated with global (shared) regularities and local (specific) dynamics. In this paper, we seek to tackle such forecasting problems with DeepDGL, a deep forecasting model that disentangles dynamics into global and local temporal patterns. DeepDGL employs an encoder-decoder architecture, consisting of two encoders to learn global and local temporal patterns, respectively, and a decoder to make mu…
Aalborg University（奥尔堡大学）；Mila-Quebec AI Institute（米拉-魁北克人工智能研究所）；HEC Montreal（蒙特利尔高等商学院）
In step with the digitalization of transportation, we are witnessing a growing range of path-based smart-city applications, e.g., travel-time estimation and travel path ranking. A temporal path(TP) that includes temporal information, e.g., departure time, into the path is fundamental to enable such applications. In this setting, it is essential to learn generic temporal path representations(TPRs) that consider spatial and temporal correlations simultaneously and that can be used in different applications, i.e., downstream tasks. Existing methods fail to achieve the goal since (i) supervised me…
National University of Singapore（新加坡国立大学）；University of Hong Kong（香港大学）；JD Intelligent Cities Research（京东智能城市研究院）；JD iCity, JD Technology（京东科技京东智联云）
Deep learning models are modern tools for spatio-temporal graph (STG) forecasting. Though successful, we argue that data scarcity is a key factor limiting their recent improvements. Meanwhile, contrastive learning has been an effective method for providing self-supervision signals and addressing data scarcity in various domains. In view of this, one may ask: can we leverage the additional signals from contrastive learning to alleviate data scarcity, so as to benefit STG forecasting? To answer this question, we present the first systematic exploration on incorporating contrastive learning into …
Learning Time-aware Graph Structures for Spatially Correlated Time Series Forecasting
Southwest Jiaotong University（西南交通大学）；East China Normal University（华东师范大学）；Aalborg University（奥尔堡大学）；University of Electronic Science and Technology of China（电子科技大学）；Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）
This paper presents TS2Vec, a universal framework for learning representations of time series in an arbitrary semantic level. Unlike existing methods, TS2Vec performs contrastive learning in a hierarchical way over augmented context views, which enables a robust contextual representation for each timestamp. Furthermore, to obtain the representation of an arbitrary sub-sequence in the time series, we can apply a simple aggregation over the representations of corresponding timestamps. We conduct extensive experiments on time series classification tasks to evaluate the quality of time series repr…
Personalized Federated Learning for Spatio-Temporal Forecasting: A Dual Semantic Alignment-Based Contrastive Approach
Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
STS-CCL: Spatial-Temporal Synchronous Contextual Contrastive Learning for Urban Traffic Forecasting
Zhejiang University（浙江大学）；South China University of Technology（华南理工大学）；The University of Sydney（悉尼大学）；Chongqing University（重庆大学）；Zhejiang Institute of Industry and Information Technology（浙江工业信息技术研究院）
University of California, San Diego（加州大学圣地亚哥分校）；SRI International（SRI国际）
Neural Ordinary Differential Equations (NODEs) have proven to be a powerful modeling tool for approximating (interpolation) and forecasting (extrapolation) irregularly sampled time series data. However, their performance degrades substantially when applied to real-world data, especially long-term data with complex behaviors (e.g., long-term trend across years, mid-term seasonality across months, and short-term local variation across days). To address the modeling of such complex data with different behaviors at different frequencies (time spans), we propose a novel progressive learning paradig…
Aligning image and text encoders from scratch using contrastive learning requires large amounts of paired image-text data. We alleviate this need by aligning individually pre-trained language and vision representation models using a much smaller amount of paired data, augmented with a curriculum learning algorithm to learn fine-grained vision-language alignments. TOnICS (Training with Ontology-Informed Contrastive Sampling) initially samples minibatches whose image-text pairs contain a wide variety of objects to learn object-level alignment, and progressively samples minibatches where all imag…
Salesforce Research Asia（赛富时亚洲研究院）；Singapore Management University（新加坡管理大学）
Deep learning has been actively studied for time series forecasting, and the mainstream paradigm is based on the end-to-end training of neural network architectures, ranging from classical LSTM/RNNs to more recent TCNs and Transformers. Motivated by the recent success of representation learning in computer vision and natural language processing, we argue that a more promising paradigm for time series forecasting, is to first learn disentangled feature representations, followed by a simple regression fine-tuning step -- we justify such a paradigm from a causal perspective. Following this princi…
