This paper proposes a temporal graph neural network model for forecasting of graph-structured irregularly observed time series. Our TGNN4I model is designed to handle both irregular time steps and partial observations of the graph. This is achieved by introducing a time-continuous latent state in each node, following a linear Ordinary Differential Equation (ODE) defined by the output of a Gated Recurrent Unit (GRU). The ODE has an explicit solution as a combination of exponential decay and periodic dynamics. Observations in the graph neighborhood are taken into account by integrating graph neural network layers in both the GRU state update and predictive model. The time-continuous dynamics additionally enable the model to make predictions at arbitrary time steps. We propose a loss function that leverages this and allows for training the model for forecasting over different time horizons. Experiments on simulated data and real-world data from traffic and climate modeling validate the usefulness of both the graph structure and time-continuous dynamics in settings with irregular observations.
Electronic records contain sequences of events, some of which take place all at once in a single visit, and others that are dispersed over multiple visits, each with a different timestamp. We postulate that fine temporal detail, e.g., whether a series of blood tests are completed at once or in rapid succession should not alter predictions based on this data. Motivated by this intuition, we propose models for analyzing sequences of multivariate clinical time series data that are invariant to this temporal clustering. We propose an efficient data augmentation technique that exploits the postulat…
A practical approach to evaluate performance of a Gaussian process regression models (GPR) for irregularly sampled sparse time-series is introduced. The approach entails construction of a secondary autoregressive model using the fine scale predictions to forecast a future observation used in GPR. We build different GPR models for Ornstein-Uhlenbeck and Fractional processes for simulated toy data with different sparsity levels to assess the utility of the approach.
Mathematical Institute, University of Oxford（牛津大学数学研究所）；The Alan Turing Institute（艾伦·图灵研究所）
Neural ordinary differential equations are an attractive option for modelling temporal dynamics. However, a fundamental issue is that the solution to an ordinary differential equation is determined by its initial condition, and there is no mechanism for adjusting the trajectory based on subsequent observations. Here, we demonstrate how this may be resolved through the well-understood mathematics of \emph{controlled differential equations}. The resulting \emph{neural controlled differential equation} model is directly applicable to the general setting of partially-observed irregularly-sampled m…
Humboldt-Universität zu Berlin（柏林洪堡大学）；Bosch Center for AI（博世人工智能中心）
Recurrent neural networks (RNNs) are a popular choice for modeling sequential data. Modern RNN architectures assume constant time-intervals between observations. However, in many datasets (e.g. medical records) observation times are irregular and can carry important information. To address this challenge, we propose continuous recurrent units (CRUs) -- a neural architecture that can naturally handle irregular intervals between observations. The CRU assumes a hidden state, which evolves according to a linear stochastic differential equation and is integrated into an encoder-decoder framework. T…
Supervised learning with irregularly sampled time series have been a challenge to Machine Learning methods due to the obstacle of dealing with irregular time intervals. Some papers introduced recently recurrent neural network models that deals with irregularity, but most of them rely on complex mechanisms to achieve a better performance. This work propose a novel method to represent timestamps (hours or dates) as dense vectors using sinusoidal functions, called Time Embeddings. As a data input method it and can be applied to most machine learning models. The method was evaluated with two predi…
College of Information and Computer Sciences（信息与计算机科学学院）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）
Irregularly sampled time series data arise naturally in many application domains including biology, ecology, climate science, astronomy, and health. Such data represent fundamental challenges to many classical models from machine learning and statistics due to the presence of non-uniform intervals between observations. However, there has been significant progress within the machine learning community over the last decade on developing specialized models and architectures for learning from irregularly sampled univariate and multivariate time series data. In this survey, we first describe severa…
Forecasting West Nile Virus with Graph Neural Networks: Harnessing Spatial Dependence in Irregularly Sampled Geospatial Data
Department of Statistics, University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校统计系）；Department of Statistics, Texas A&M University（德克萨斯A&M大学统计系）；Department of Pathobiology, University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校病理生物学系）
Irregularly-sampled time series occur in many domains including healthcare. They can be challenging to model because they do not naturally yield a fixed-dimensional representation as required by many standard machine learning models. In this paper, we consider irregular sampling from the perspective of missing data. We model observed irregularly-sampled time series data as a sequence of index-value pairs sampled from a continuous but unobserved function. We introduce an encoder-decoder framework for learning from such generic indexed sequences. We propose learning methods for this framework ba…
Many real-world systems, such as moving planets, can be considered as multi-agent dynamic systems, where objects interact with each other and co-evolve along with the time. Such dynamics is usually difficult to capture, and understanding and predicting the dynamics based on observed trajectories of objects become a critical research problem in many domains. Most existing algorithms, however, assume the observations are regularly sampled and all the objects can be fully observed at each sampling time, which is impractical for many applications. In this paper, we propose to learn system dynamics…
University of Toronto（多伦多大学）；Vector Institute（向量研究所）
Time series with non-uniform intervals occur in many applications, and are difficult to model using standard recurrent neural networks (RNNs). We generalize RNNs to have continuous-time hidden dynamics defined by ordinary differential equations (ODEs), a model we call ODE-RNNs. Furthermore, we use ODE-RNNs to replace the recognition network of the recently-proposed Latent ODE model. Both ODE-RNNs and Latent ODEs can naturally handle arbitrary time gaps between observations, and can explicitly model the probability of observation times using Poisson processes. We show experimentally that these …
Harvard University（哈佛大学）；University of Ljubljana（卢布尔雅那大学）；MIT Lincoln Laboratory（麻省理工学院林肯实验室）
In many domains, including healthcare, biology, and climate science, time series are irregularly sampled with varying time intervals between successive readouts and different subsets of variables (sensors) observed at different time points. Here, we introduce RAINDROP, a graph neural network that embeds irregularly sampled and multivariate time series while also learning the dynamics of sensors purely from observational data. RAINDROP represents every sample as a separate sensor graph and models time-varying dependencies between sensors with a novel message passing operator. It estimates the l…
Inverse problems involve making inference about unknown parameters of a physical process using observational data. This paper investigates an important class of inverse problems -- the estimation of the initial condition of a spatio-temporal advection-diffusion process using spatially sparse data streams. Three spatial sampling schemes are considered, including irregular, non-uniform and shifted uniform sampling. The irregular sampling scheme is the general scenario, while computationally efficient solutions are available in the spectral domain for non-uniform and shifted uniform sampling. For…
University College London（伦敦大学学院）；Biomediq A/S（Biomediq公司）；University of Copenhagen（哥本哈根大学）；Cerebriu A/S（Cerebriu公司）
Learning temporal patterns from multivariate longitudinal data is challenging especially in cases when data is sporadic, as often seen in, e.g., healthcare applications where the data can suffer from irregularity and asynchronicity as the time between consecutive data points can vary across features and samples, hindering the application of existing deep learning models that are constructed for complete, evenly spaced data with fixed sequence lengths. In this paper, a novel deep learning-based model is developed for modeling multiple temporal features in sporadic data using an integrated deep …
RBF-MGN:Solving spatiotemporal PDEs with Physics-informed Graph Neural Network
College of Aerospace Science and Engineering, National University of Defense Technology（国防科技大学航天科学与工程学院）；National Innovation Institute of Defense Technology, Chinese Academy of Military Science（中国人民解放军军事科学院国防科技创新研究院）
Time series observations are ubiquitous in astronomy, and are generated to distinguish between different types of supernovae, to detect and characterize extrasolar planets and to classify variable stars. These time series are usually modeled using a parametric and/or physical model that assumes independent and homoscedastic errors, but in many cases these assumptions are not accurate and there remains a temporal dependency structure on the errors. This can occur, for example, when the proposed model cannot explain all the variability of the data or when the parameters of the model are not prop…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；IBM T.J. Watson Research Center（IBM托马斯·J·沃森研究中心）；NVIDIA（英伟达）
With the increasing adoption of graph neural networks (GNNs) in the machine learning community, GPUs have become an essential tool to accelerate GNN training. However, training GNNs on very large graphs that do not fit in GPU memory is still a challenging task. Unlike conventional neural networks, mini-batching input samples in GNNs requires complicated tasks such as traversing neighboring nodes and gathering their feature values. While this process accounts for a significant portion of the training time, we find existing GNN implementations using popular deep neural network (DNN) libraries su…
For temporal regularly spaced datasets, a lot of methods are available and the properties of these methods are extensively investigated. Less research has been performed on irregular temporal datasets subject to measurement error with complex dependence structures, while this type of datasets is widely available. In this paper, the performance of kernel smoother for trend and its derivatives is considered under dependent measurement errors and irregularly spaced sampling scheme. The error processes are assumed to be subordinated Gaussian long memory processes and have varying marginal distribu…
Machine learning methods such as convolutional neural networks (CNNs) are becoming an integral part of scientific research in many disciplines, spatial vector data often fail to be analyzed using these powerful learning methods because of its irregularities. With the aid of graph Fourier transform and convolution theorem, it is possible to convert the convolution as a point-wise product in Fourier domain and construct a learning architecture of CNN on graph for the analysis task of irregular spatial data. In this study, we used the classification task of building patterns as a case study to te…
The understanding of geographical reality is a process of data representation and pattern discovery. Former studies mainly adopted continuous-field models to represent spatial variables and to investigate the underlying spatial continuity/heterogeneity in the regular spatial domain. In this article, we introduce a more generalized model based on graph convolutional neural networks (GCNs) that can capture the complex parameters of spatial patterns underlying graph-structured spatial data, which generally contain both Euclidean spatial information and non-Euclidean feature information. A trainab…
In many fields observations are performed irregularly along time, due to either measurement limitations or lack of a constant immanent rate. While discrete-time Markov models (as Dynamic Bayesian Networks) introduce either inefficient computation or an information loss to reasoning about such processes, continuous-time Markov models assume either a discrete state space (as Continuous-Time Bayesian Networks), or a flat continuous state space (as stochastic differential equations). To address these problems, we present a new modeling class called Irregular-Time Bayesian Networks (ITBNs), general…
University of Rochester（罗切斯特大学）；Stony Brook University（石溪大学）
The marriage of recurrent neural networks and neural ordinary differential networks (ODE-RNN) is effective in modeling irregularly-observed sequences. While ODE produces the smooth hidden states between observation intervals, the RNN will trigger a hidden state jump when a new observation arrives, thus cause the interpolation discontinuity problem. To address this issue, we propose the cubic spline smoothing compensation, which is a stand-alone module upon either the output or the hidden state of ODE-RNN and can be trained end-to-end. We derive its analytical solution and provide its theoretic…
University of California, San Diego（加利福尼亚大学圣地亚哥分校）；Qualcomm AI Research（高通人工智能研究院）；Duke University（杜克大学）
Recent advances demonstrate that irregularly wired neural networks from Neural Architecture Search (NAS) and Random Wiring can not only automate the design of deep neural networks but also emit models that outperform previous manual designs. These designs are especially effective while designing neural architectures under hard resource constraints (memory, MACs, . . . ) which highlights the importance of this class of designing neural networks. However, such a move creates complication in the previously streamlined pattern of execution. In fact one of the main challenges is that the order of s…
Toward Temporal Causal Representation Learning with Tensor Decomposition
Department of Mechanical & Industrial Engineering, Northeastern University, Boston, MA, USA（机械与工业工程系，东北大学）；Department of Industrial and Systems Engineering, Lehigh University, Bethlehem, PA, USA（工业与系统工程系，莱斯大学）；Department of Industrial and Systems Engineering, University of Florida, Gainesville, FL, USA（工业与系统工程系，佛罗里达大学）
College of Information and Computer Sciences（信息与计算机科学学院）；University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）
We present a general framework for classification of sparse and irregularly-sampled time series. The properties of such time series can result in substantial uncertainty about the values of the underlying temporal processes, while making the data difficult to deal with using standard classification methods that assume fixed-dimensional feature spaces. To address these challenges, we propose an uncertainty-aware classification framework based on a special computational layer we refer to as the Gaussian process adapter that can connect irregularly sampled time series data to any black-box classi…
University of Modena and Reggio Emilia（摩德纳-雷焦艾米利亚大学）
We present a novel and hierarchical approach for supervised classification of signals spanning over a fixed graph, reflecting shared properties of the dataset. To this end, we introduce a Convolutional Cluster Pooling layer exploiting a multi-scale clustering in order to highlight, at different resolutions, locally connected regions on the input graph. Our proposal generalises well-established neural models such as Convolutional Neural Networks (CNNs) on irregular and complex domains, by means of the exploitation of the weight sharing property in a graph-oriented architecture. In this work, su…
Solving Irregular and Data-enriched Differential Equations using Deep Neural Networks
Oden Institute for Computational Engineering and Sciences, University of Texas at Austin, Austin,TX 78712（计算工程与科学研究院，德克萨斯大学奥斯汀分校，奥斯汀，TX 78712）；Department of Astronomy, The University of Texas at Austin, Austin, TX 78712（天文学系，德克萨斯大学奥斯汀分校，奥斯汀，TX 78712）；Institute for Fusion Studies, University of Texas at Austin, Austin,TX 78712（融合研究学院，德克萨斯大学奥斯汀分校，奥斯汀，TX 78712）
