The identification of nonlinear dynamics from observations is essential for the alignment of the theoretical ideas and experimental data. The last, in turn, is often corrupted by the side effects and noise of different natures, so probabilistic approaches could give a more general picture of the process. At the same time, high-dimensional probabilities modeling is a challenging and data-intensive task. In this paper, we establish a parallel between the dynamical systems modeling and language modeling tasks. We propose a transformer-based model that incorporates geometrical properties of the data and provide an iterative training algorithm allowing the fine-grid approximation of the conditional probabilities of high-dimensional dynamical systems.
College of Intelligence and Computing, Tianjin University（天津大学智能与计算学部）；Microsoft Research Asia（微软亚洲研究院）；School of Computer Science and Technology, Beijing Institute of Technology（北京理工大学计算机学院）
Latest development of neural models has connected the encoder and decoder through a self-attention mechanism. In particular, Transformer, which is solely based on self-attention, has led to breakthroughs in Natural Language Processing (NLP) tasks. However, the multi-head attention mechanism, as a key component of Transformer, limits the effective deployment of the model to a resource-limited setting. In this paper, based on the ideas of tensor decomposition and parameters sharing, we propose a novel self-attention model (namely Multi-linear attention) with Block-Term Tensor Decomposition (BTD)…
Maastricht University（马斯特里赫特大学）；Mathematics Centre Maastricht（马斯特里赫特数学中心）
Reliable weather forecasting is of great importance in science, business, and society. The best performing data-driven models for weather prediction tasks rely on recurrent or convolutional neural networks, where some of which incorporate attention mechanisms. In this work, we introduce a novel model based on Transformer architecture for weather forecasting. The proposed Tensorial Encoder Transformer (TENT) model is equipped with tensorial attention and thus it exploits the spatiotemporal structure of weather data by processing it in multidimensional tensorial format. We show that compared to …
Optimality and NP-Hardness of Transformers in Learning Markovian Dynamical Functions
Department of Computer Science, Rensselaer Polytechnic Institute（拉特格斯理工学院计算机科学系）；Department of Computer Science and Engineering, The Chinese University of Hong Kong（香港中文大学计算机科学与工程系）；Mathematics of Computation, IBM Research（IBM研究院数学部）
The Transformer and its variants have been proven to be efficient sequence learners in many different domains. Despite their staggering success, a critical issue has been the enormous number of parameters that must be trained (ranging from $10^7$ to $10^{11}$) along with the quadratic complexity of dot-product attention. In this work, we investigate the problem of approximating the two central components of the Transformer -- multi-head self-attention and point-wise feed-forward transformation, with reduced parameter space and computational complexity. We build upon recent developments in anal…
Technical University of Denmark（丹麦技术大学）；Demant A/S（丹麦德玛特公司）；KU Leuven（鲁汶大学）
Multi-head self-attention forms the core of Transformer networks. However, their quadratically growing complexity with respect to the input sequence length impedes their deployment on resource-constrained edge devices. We address this challenge by proposing a dynamic pruning method, which exploits the temporal stability of data across tokens to reduce inference cost. The threshold-based method only retains significant differences between the subsequent tokens, effectively reducing the number of multiply-accumulates, as well as the internal tensor data sizes. The approach is evaluated on the Go…
We introduce a new way of learning to encode position information for non-recurrent models, such as Transformer models. Unlike RNN and LSTM, which contain inductive bias by loading the input tokens sequentially, non-recurrent models are less sensitive to position. The main reason is that position information among input units is not inherently encoded, i.e., the models are permutation equivalent; this problem justifies why all of the existing models are accompanied by a sinusoidal encoding/embedding layer at the input. However, this solution has clear limitations: the sinusoidal encoding is no…
Transformers are widely used in natural language processing due to their ability to model longer-term dependencies in text. Although these models achieve state-of-the-art performance for many language related tasks, their applicability outside of the natural language processing field has been minimal. In this work, we propose the use of transformer models for the prediction of dynamical systems representative of physical phenomena. The use of Koopman based embeddings provide a unique and powerful method for projecting any dynamical system into a vector representation which can then be predicte…
Ghent University（根特大学）；University of Ottawa（渥太华大学）；Vector Institute for Artificial Intelligence（向量人工智能研究所）；Lawrence Berkeley National Laboratory（劳伦斯伯克利国家实验室）
We show that a neural network originally designed for language processing can learn the dynamical rules of a stochastic system by observation of a single dynamical trajectory of the system, and can accurately predict its emergent behavior under conditions not observed during training. We consider a lattice model of active matter undergoing continuous-time Monte Carlo dynamics, simulated at a density at which its steady state comprises small, dispersed clusters. We train a neural network called a transformer on a single trajectory of the model. The transformer, which we show has the capacity to…
OT-Transformer: A Continuous-time Transformer Architecture with Optimal Transport Regularization
University of California, Los Angeles（加利福尼亚大学洛杉矶分校）；Oden Institute for Computational Engineering and Sciences, University of Texas, Austin（德克萨斯大学奥斯汀分校奥登计算工程与科学学院）
DeepKoopFormer: A Koopman Enhanced Transformer Based Architecture for Time Series Forecasting
Helmholtz Center for Environmental Research-UFZ（海德堡环境研究中心-UFZ）；Delft Center for Systems and Control（代尔夫特系统与控制中心）；School of Engineering, University of Pittsburgh（匹兹堡大学工程学院）
Deconstructing Recurrence, Attention, and Gating: Investigating the transferability of Transformers and Gated Recurrent Neural Networks in forecasting of dynamical systems
School of Engineering and Applied Sciences, Harvard University（哈佛大学工程与应用科学学院）；Institute of Structural Engineering, ETH Zurich（苏黎世联邦理工学院结构工程研究所）
Volume-Preserving Transformers for Learning Time Series Data with Structure
Max-Planck-Institut für Plasmaphysik（马克斯·普朗克等离子体物理研究所）；ONERA（法国航空航天研究院）；Université Paris Saclay（巴黎萨克雷大学）；Arts et Métiers Institute of Technology（国立巴黎高等机械学院）；Zentrum Mathematik, Technische Universität München（慕尼黑工业大学数学中心）
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究所）
Proper states' representations are the key to the successful dynamics modeling of chaotic systems. Inspired by recent advances of deep representations in various areas such as natural language processing and computer vision, we propose the adaptation of the state-of-art Transformer model in application to the dynamical systems modeling. The model demonstrates promising results in trajectories generation as well as in the general attractors' characteristics approximation, including states' distribution and Lyapunov exponent.
Machine Learning (ML) for information security (InfoSec) utilizes distinct data types and formats which require different treatments during optimization/training on raw data. In this paper, we implement a malicious/benign URL predictor based on a transformer architecture that is trained from scratch. We show that in contrast to conventional natural language processing (NLP) transformers, this model requires a different training approach to work well. Specifically, we show that 1) pre-training on a massive corpus of unlabeled URL data for an auto-regressive task does not readily transfer to mal…
T-SHRED: Symbolic Regression for Regularization and Model Discovery with Transformer Shallow Recurrent Decoders
Electrical and Computer Engineering, University of Washington（华盛顿大学电气与计算机工程系）；Applied Mathematics, University of Washington（华盛顿大学应用数学系）；Computer Science & Engineering, University of Washington（华盛顿大学计算机科学与工程系）
A novel forecasting framework combining virtual samples and enhanced Transformer models for tourism demand forecasting
School of Computer, Electronics and Information, Guangxi University（广西大学计算机与电子信息学院）；School of Electrical Engineering, Guangxi University（广西大学电气工程学院）；School of Mathematics and Statistics, Xuzhou University of Technology（徐州工程学院数学与统计学院）
Efficient Large-Scale Traffic Forecasting with Transformers: A Spatial Data Management Perspective
University of Electronic Science and Technology of China（电子科技大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Auburn University（奥本大学）；Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）；National University of Singapore（新加坡国立大学）；Peking University（北京大学）；Yangtze Delta Region Institute (Quzhou)（长三角研究院（衢州））
Entropy Guided Dynamic Patch Segmentation for Time Series Transformers
School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore（南洋理工大学电子与电气工程学院）；Institute for Infocomm Research, A*STAR, Singapore（信息通信研究所，A*STAR，新加坡）；Department of Computer Science, Khalifa University, UAE（计算机科学系，哈利法大学，阿联酋）；Information Systems Technology and Design, Singapore University of Technology and Design, Singapore（信息系统技术与设计，新加坡科技与设计大学，新加坡）
