Tabular datasets are ubiquitous in data science applications. Given their importance, it seems natural to apply state-of-the-art deep learning algorithms in order to fully unlock their potential. Here we propose neural network models that represent tabular time series that can optionally leverage their hierarchical structure. This results in two architectures for tabular time series: one for learning representations that is analogous to BERT and can be pre-trained end-to-end and used in downstream tasks, and one that is akin to GPT and can be used for generation of realistic synthetic tabular sequences. We demonstrate our models on two datasets: a synthetic credit card transaction dataset, where the learned representations are used for fraud detection and synthetic data generation, and on a real pollution dataset, where the learned encodings are used to predict atmospheric pollutant concentrations. Code and data are available at https://github.com/IBM/TabFormer.
HTMformer: Hybrid Time and Multivariate Transformer for Time Series Forecasting
Northwestern Polytechnical University（北华理工大学）；MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University（人工智能教育部重点实验室，上海交通大学AI研究院）；Ningbo Institute of Digital Twin, Eastern Institute of Technology（宁波数字孪生研究所，东部技术研究所）；Ningbo Key Laboratory of Spatial Intelligence and Digital Derivative（宁波空间智能与数字衍生关键实验室）
Ister: Linear Transformer for Efficient Multivariate Time Series Forecasting
Hong Kong University of Science and Technology (Guangzhou)（香港理工大学（广州））；ShenZhen University（深圳大学）；Shenzhen Energy Group Co., Ltd.（深圳能源集团有限公司）；South China University of Technology（华南理工大学）
SCFormer: Structured Channel-wise Transformer with Cumulative Historical State for Multivariate Time Series Forecasting
Xinjiang Technical Institute of Physics and Chemistry, Chinese Academy of Sciences（新疆物理化学技术研究所，中国科学院）；University of Chinese Academy of Sciences（中国科学院大学）；Xinjiang Laboratory of Minority Speech and Language Information Processing（新疆少数民族语言信息处理实验室）
HDT: Hierarchical Discrete Transformer for Multivariate Time Series Forecasting
College of Computing and Data Science, Nanyang Technological University (NTU)（南洋理工大学计算与数据科学学院）；Webank-NTU Joint Research Institute on Fintech, NTU（微众银行-南洋理工大学金融科技联合研究院）；Tencent AI Lab（腾讯人工智能实验室）
VCformer: Variable Correlation Transformer with Inherent Lagged Correlation for Multivariate Time Series Forecasting
College of Computer Science and Software Engineering, Shenzhen University（深圳大学计算机与软件学院）；National Engineering Laboratory for Big Data System Computing Technology, Shenzhen University（深圳大学国家大数据系统计算技术工程实验室）
Multivariate time series forecasting (MTSF) is a fundamental problem in numerous real-world applications. Recently, Transformer has become the de facto solution for MTSF, especially for the long-term cases. However, except for the one forward operation, the basic configurations in existing MTSF Transformer architectures were barely carefully verified. In this study, we point out that the current tokenization strategy in MTSF Transformer architectures ignores the token uniformity inductive bias of Transformers. Therefore, the vanilla MTSF transformer struggles to capture details in time series …
Radiant First Research Limited（Radiant First研究有限公司）；The University of Hong Kong（香港大学）；Koi Investment Partners International Limited（Koi国际投资伙伴有限公司）
A reliable and efficient representation of multivariate time series is crucial in various downstream machine learning tasks. In multivariate time series forecasting, each variable depends on its historical values and there are inter-dependencies among variables as well. Models have to be designed to capture both intra- and inter-relationships among the time series. To move towards this goal, we propose the Time Series Attention Transformer (TSAT) for multivariate time series representation learning. Using TSAT, we represent both temporal information and inter-dependencies of multivariate time …
BioSensics LLC（BioSensics有限责任公司）；Brown University（布朗大学）；University of Southern California（南加州大学）
Epilepsy is one of the most common neurological disorders, typically observed via seizure episodes. Epileptic seizures are commonly monitored through electroencephalogram (EEG) recordings due to their routine and low expense collection. The stochastic nature of EEG makes seizure identification via manual inspections performed by highly-trained experts a tedious endeavor, motivating the use of automated identification. The literature on automated identification focuses mostly on supervised learning methods requiring expert labels of EEG segments that contain seizures, which are difficult to obt…
Imperial College London（帝国理工学院）；Loughborough University（拉夫堡大学）
Efficient anomaly detection and diagnosis in multivariate time-series data is of great importance for modern industrial applications. However, building a system that is able to quickly and accurately pinpoint anomalous observations is a challenging problem. This is due to the lack of anomaly labels, high data volatility and the demands of ultra-low inference times in modern applications. Despite the recent developments of deep learning approaches for anomaly detection, only a few of them can address all of these challenges. In this paper, we propose TranAD, a deep transformer network based ano…
Multivariate time-series data are frequently observed in critical care settings and are typically characterized by sparsity (missing information) and irregular time intervals. Existing approaches for learning representations in this domain handle these challenges by either aggregation or imputation of values, which in-turn suppresses the fine-grained information and adds undesirable noise/overhead into the machine learning model. To tackle this problem, we propose a Self-supervised Transformer for Time-Series (STraTS) model which overcomes these pitfalls by treating time-series as a set of obs…
Low Rank Transformer for Multivariate Time Series Anomaly Detection and Localization
KIOS Research and Innovation Center of Excellence（KIOS研究与创新中心卓越研究所）；Department of Electrical and Computer Engineering, University of Cyprus（电子与计算机工程系，塞浦路斯大学）；Department of Mathematics and Statistics, University of Cyprus（数学与统计学系，塞浦路斯大学）
George Washington University（乔治华盛顿大学）；Rutgers University（罗格斯大学）；Shandong University（山东大学）
Many real-world IoT systems, which include a variety of internet-connected sensory devices, produce substantial amounts of multivariate time series data. Meanwhile, vital IoT infrastructures like smart power grids and water distribution networks are frequently targeted by cyber-attacks, making anomaly detection an important study topic. Modeling such relatedness is, nevertheless, unavoidable for any efficient and effective anomaly detection system, given the intricate topological and nonlinear connections that are originally unknown among sensors. Furthermore, detecting anomalies in multivaria…
MCformer: Multivariate Time Series Forecasting with Mixed-Channels Transformer
School of Computer Science, University of South China（南华大学计算机学院）；School of Computer Science and Technology, Dalian University of Technology（大连理工大学计算机科学与技术学院）；Department of Computer & Communication Engineering, University of Science and Technology Beijing（北京科技大学计算机与通信工程学院）
One Transformer for All Time Series: Representing and Training with Time-Dependent Heterogeneous Tabular Data
Prometeia Spa（Prometeia公司）；Prometeia Associazione（Prometeia协会）；University of Modena and Reggio Emilia（摩德纳和雷焦艾米利亚大学）；University of Padova（帕多瓦大学）；Istituto di Informatica e Telematica CNR（意大利国家研究委员会信息与远程信息处理研究所）
Airt Research（Airt研究中心）；Algebra University College（代数大学学院）
Time-series classification is one of the most frequently performed tasks in industrial data science, and one of the most widely used data representation in the industrial setting is tabular representation. In this work, we propose a novel scalable architecture for learning representations from tabular time-series data and subsequently performing downstream tasks such as time-series classification. The representation learning framework is end-to-end, akin to bidirectional encoder representations from transformers (BERT) in language modeling, however, we introduce novel masking technique suitabl…
