Crowd flow prediction has been increasingly investigated in intelligent urban computing field as a fundamental component of urban management system. The most challenging part of predicting crowd flow is to measure the complicated spatial-temporal dependencies. A prevalent solution employed in current methods is to divide and conquer the spatial and temporal information by various architectures (e.g., CNN/GCN, LSTM). However, this strategy has two disadvantages: (1) the sophisticated dependencies are also divided and therefore partially isolated; (2) the spatial-temporal features are transformed into latent representations when passing through different architectures, making it hard to interpret the predicted crowd flow. To address these issues, we propose a Spatial-Temporal Self-Attention Network (STSAN) with an ST encoding gate that calculates the entire spatial-temporal representation with positional and time encodings and therefore avoids dividing the dependencies. Furthermore, we develop a Multi-aspect attention mechanism that applies scaled dot-product attention over spatial-temporal information and measures the attention weights that explicitly indicate the dependencies. Experimental results on traffic and mobile data demonstrate that the proposed method reduces inflow and outflow RMSE by 16% and 8% on the Taxi-NYC dataset compared to the SOTA baselines.
Sun Yat-sen University（中山大学）；The Chinese University of Hong Kong（香港中文大学）；Beihang University（北京航空航天大学）
Traffic flow prediction is crucial for urban traffic management and public safety. Its key challenges lie in how to adaptively integrate the various factors that affect the flow changes. In this paper, we propose a unified neural network module to address this problem, called Attentive Crowd Flow Machine~(ACFM), which is able to infer the evolution of the crowd flow by learning dynamic representations of temporally-varying data with an attention mechanism. Specifically, the ACFM is composed of two progressive ConvLSTM units connected with a convolutional layer for spatial weight prediction. Th…
University of British Columbia（不列颠哥伦比亚大学）；Tsinghua University（清华大学）
Inspired by the observation that humans are able to process videos efficiently by only paying attention where and when it is needed, we propose an interpretable and easy plug-in spatial-temporal attention mechanism for video action recognition. For spatial attention, we learn a saliency mask to allow the model to focus on the most salient parts of the feature maps. For temporal attention, we employ a convolutional LSTM based attention mechanism to identify the most relevant frames from an input video. Further, we propose a set of regularizers to ensure that our attention mechanism attends to c…
Interpretable Kolmogorov-Arnold Network with Feature-Isolated Temporal Attention Mechanism for Electricity Load Forecasting
Department of Data Science and AI, Faculty of Information Technology, Monash University（蒙纳士大学信息技术学院数据科学与人工智能系）；Monash Energy Institute, Monash University（蒙纳士大学蒙纳士能源研究所）
Motional（莫雄公司）；Boston University（波士顿大学）；Iowa State University（爱荷华州立大学）
Autonomous vehicles must balance a complex set of objectives. There is no consensus on how they should do so, nor on a model for specifying a desired driving behavior. We created a dataset to help address some of these questions in a limited operating domain. The data consists of 92 traffic scenarios, with multiple ways of traversing each scenario. Multiple annotators expressed their preference between pairs of scenario traversals. We used the data to compare an instance of a rulebook, carefully hand-crafted independently of the dataset, with several interpretable machine learning models such …
Microsoft Research（微软研究院）；Southwest Jiaotong University（西南交通大学）；Xidian University（西安电子科技大学）；Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院）
Forecasting the flow of crowds is of great importance to traffic management and public safety, yet a very challenging task affected by many complex factors, such as inter-region traffic, events and weather. In this paper, we propose a deep-learning-based approach, called ST-ResNet, to collectively forecast the in-flow and out-flow of crowds in each and every region through a city. We design an end-to-end structure of ST-ResNet based on unique properties of spatio-temporal data. More specifically, we employ the framework of the residual neural networks to model the temporal closeness, period, a…
University of Pisa（比萨大学）；Free University of Bolzano（博尔扎诺自由大学）；Bruno Kessler Foundation（布鲁诺·凯斯勒基金会）；Institute of Information Science and Technology (ISTI), National Research Council (CNR)（国家研究委员会信息科学与技术研究所）
Thanks to the diffusion of the Internet of Things, nowadays it is possible to sense human mobility almost in real time using unconventional methods (e.g., number of bikes in a bike station). Due to the diffusion of such technologies, the last years have witnessed a significant growth of human mobility studies, motivated by their importance in a wide range of applications, from traffic management to public security and computational epidemiology. A mobility task that is becoming prominent is crowd flow prediction, i.e., forecasting aggregated incoming and outgoing flows in the locations of a ge…
SenseTime Research（商汤科技研究院）；National University of Singapore（新加坡国立大学）
Understanding crowd motion dynamics is critical to real-world applications, e.g., surveillance systems and autonomous driving. This is challenging because it requires effectively modeling the socially aware crowd spatial interaction and complex temporal dependencies. We believe attention is the most important factor for trajectory prediction. In this paper, we present STAR, a Spatio-Temporal grAph tRansformer framework, which tackles trajectory prediction by only attention mechanisms. STAR models intra-graph crowd interaction by TGConv, a novel Transformer-based graph convolution mechanism. Th…
Nanyang Technological University（南洋理工大学）；University of California, Davis（加州大学戴维斯分校）
Machine learning-based methods have achieved successful applications in machinery fault diagnosis. However, the main limitation that exists for these methods is that they operate as a black box and are generally not interpretable. This paper proposes a novel neural network structure, called temporal logic neural network (TLNN), in which the neurons of the network are logic propositions. More importantly, the network can be described and interpreted as a weighted signal temporal logic. TLNN not only keeps the nice properties of traditional neuron networks but also provides a formal interpretati…
University of Oxford（牛津大学）；Google Cloud AI Research（谷歌云人工智能研究院）；Google Cloud AI（谷歌云人工智能）
Multi-horizon forecasting problems often contain a complex mix of inputs -- including static (i.e. time-invariant) covariates, known future inputs, and other exogenous time series that are only observed historically -- without any prior information on how they interact with the target. While several deep learning models have been proposed for multi-step prediction, they typically comprise black-box models which do not account for the full range of inputs present in common scenarios. In this paper, we introduce the Temporal Fusion Transformer (TFT) -- a novel attention-based architecture which …
Rensselaer Polytechnic Institute（伦斯勒理工学院）；IBM T.J. Watson Research Center（IBM托马斯·J·沃森研究中心）；IBM Research（IBM研究院）
Most existing Time series classification (TSC) models lack interpretability and are difficult to inspect. Interpretable machine learning models can aid in discovering patterns in data as well as give easy-to-understand insights to domain specialists. In this study, we present Neuro-Symbolic Time Series Classification (NSTSC), a neuro-symbolic model that leverages signal temporal logic (STL) and neural network (NN) to accomplish TSC tasks using multi-view data representation and expresses the model as a human-readable, interpretable formula. In NSTSC, each neuron is linked to a symbolic express…
University of Southern California（南加州大学）；University of Trieste（的里雅斯特大学）
The Internet-of-Things, complex sensor networks, multi-agent cyber-physical systems are all examples of spatially distributed systems that continuously evolve in time. Such systems generate huge amounts of spatio-temporal data, and system designers are often interested in analyzing and discovering structure within the data. There has been considerable interest in learning causal and logical properties of temporal data using logics such as Signal Temporal Logic (STL); however, there is limited work on discovering such relations on spatio-temporal data. We propose the first set of algorithms for…
Human visual attention is a complex phenomenon. A computational modeling of this phenomenon must take into account where people look in order to evaluate which are the salient locations (spatial distribution of the fixations), when they look in those locations to understand the temporal development of the exploration (temporal order of the fixations), and how they move from one location to another with respect to the dynamics of the scene and the mechanics of the eyes (dynamics). State-of-the-art models focus on learning saliency maps from human data, a process that only takes into account the…
The Pennsylvania State University（宾夕法尼亚州立大学）；Duke University（杜克大学）
In many scientific problems such as video surveillance, modern genomics, and finance, data are often collected from diverse measurements across time that exhibit time-dependent heterogeneous properties. Thus, it is important to not only integrate data from multiple sources (called multi-view data), but also to incorporate time dependency for deep understanding of the underlying system. We propose a generative model based on variational autoencoder and a recurrent neural network to infer the latent dynamics for multi-view temporal data. This approach allows us to identify the disentangled laten…
TransFeat-TPP: An Interpretable Deep Covariate Temporal Point Processes
Data Science Institute, University of Technology Sydney（悉尼科技大学数据科学学院）；School of Computing, Macquarie University（麦考瑞大学计算学院）；Center for Applied Statistics and School of Statistics, Renmin University of China（中国人民大学应用统计中心与统计学院）；Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing（北京未来区块链与隐私计算高精尖创新中心）
Towards Interpretable and Trustworthy Time Series Reasoning: A BlueSky Vision
School of Computing University of Connecticut Storrs, CT（计算学院 美国康涅狄格大学 斯托尔斯分校）；Department of Machine Learning Research Morgan Stanley New York, NY（机器学习研究部 花旗集团 新 York）
JD Intelligent Cities Research（京东智能城市研究院）；JD Intelligent Cities Business Unit, JD Digits（京东数字科技智能城市业务部）；Institute of Artificial Intelligence, Southwest Jiaotong University（西南交通大学人工智能研究院）；Xidian University（西安电子科技大学）；National University of Singapore（新加坡国立大学）
Being able to predict the crowd flows in each and every part of a city, especially in irregular regions, is strategically important for traffic control, risk assessment, and public safety. However, it is very challenging because of interactions and spatial correlations between different regions. In addition, it is affected by many factors: i) multiple temporal correlations among different time intervals: closeness, period, trend; ii) complex external influential factors: weather, events; iii) meta features: time of the day, day of the week, and so on. In this paper, we formulate crowd flow for…
Clinically Interpretable Sepsis Early Warning via LLM-Guided Simulation of Temporal Physiological Dynamics
School of Electrical and Information Engineering, Tianjin University（天津大学电气与信息工程学院）；Department of Information Engineering and Computer Science, University of Trento（特伦托大学信息工程与计算机科学系）；Department of Orthopedics, Affiliated Kunshan Hospital of Jiangsu University（江苏大学附属昆山医院骨科部）；Tianjin University Chest Hospital（天津大学胸科医院）；Haihe Hospital, Tianjin University（天津大学海河医院）；Key Laboratory of Education Blockchain and Intelligent Technology, Ministry of Education, Guangxi Normal University（广西师范大学教育区块链与智能技术重点实验室）
Bangladesh University of Engineering and Technology（孟加拉国工程技术大学）
Accuracy and interpretability are two essential properties for a crime prediction model. Because of the adverse effects that the crimes can have on human life, economy and safety, we need a model that can predict future occurrence of crime as accurately as possible so that early steps can be taken to avoid the crime. On the other hand, an interpretable model reveals the reason behind a model's prediction, ensures its transparency and allows us to plan the crime prevention steps accordingly. The key challenge in developing the model is to capture the non-linear spatial dependency and temporal p…
In this paper we propose a novel human-centered approach for detecting forgery in face images, using dynamic prototypes as a form of visual explanations. Currently, most state-of-the-art deepfake detections are based on black-box models that process videos frame-by-frame for inference, and few closely examine their temporal inconsistencies. However, the existence of such temporal artifacts within deepfake videos is key in detecting and explaining deepfakes to a supervising human. To this end, we propose Dynamic Prototype Network (DPNet) -- an interpretable and effective solution that utilizes …
Bosch Center for Artificial Intelligence（博世人工智能中心）；University of Bonn（波恩大学）
The performance of autonomous systems heavily relies on their ability to generate a robust representation of the environment. Deep neural networks have greatly improved vision-based perception systems but still fail in challenging situations, e.g. sensor outages or heavy weather. These failures are often introduced by data-inherent perturbations, which significantly reduce the information provided to the perception system. We propose a functionally modularized temporal filter, which stabilizes an abstract feature representation of a single-frame segmentation model using information of previous…
Interpretable Spatio-Temporal Embedding for Brain Structural-Effective Network with Ordinary Differential Equation
University of Texas Rio Grande Valley（德克萨斯大学里奥格兰德河谷分校）；University of Maryland（马里兰大学）；University of Pittsburgh（匹兹堡大学）；Texas A&M University - Corpus Christi（德克萨斯农工大学科珀斯克里斯蒂分校）；Emory University（埃默里大学）；Lehigh University（利哈伊大学）；University of Illinois at Chicago（芝加哥伊利诺伊大学）；University of Southern California（南加州大学）
This thesis focuses on data that has complex spatio-temporal structure and on probabilistic graphical models that learn the structure in an interpretable and scalable manner. We target two research areas of interest: Gaussian graphical models for tensor-variate data and summarization of complex time-varying texts using topic models. This work advances the state-of-the-art in several directions. First, it introduces a new class of tensor-variate Gaussian graphical models via the Sylvester tensor equation. Second, it develops an optimization technique based on a fast-converging proximal alternat…
Bridging Graph Structure and Knowledge-Guided Editing for Interpretable Temporal Knowledge Graph Reasoning
Department of Data Science and Artificial Intelligence, The Hong Kong Polytechnic University（数据科学与人工智能系，香港理工大学）；School of Cybersecurity, Northwestern Polytechnical University（网络安全学院，西北工业大学）；Department of Electronic Engineering, Tsinghua University（电子工程系，清华大学）；School of Mechanical Engineering, Northwestern Polytechnical University（机械工程学院，西北工业大学）；School of Mathematics and Statistics, Xi’an Jiaotong University（数学与统计学院，西安交通大学）
SuperMAN: Interpretable and Expressive Networks over Temporally Sparse Heterogeneous Data
Meta；Center of Excellence for Molecular Prediction of IBD (PREDICT)（分子预测IBD中心（PREDICT））；Department of Clinical Medicine, Aalborg University（临床医学系，奥胡斯大学）；Department of Gastroenterology & Hepatology, Aalborg University Hospital（消化内科与肝病科，奥胡斯大学医院）；University of Copenhagen（哥本哈根大学）；Imperial College London（伦敦帝国理工学院）；Department of Biomedical Engineering, Tel-Aviv University（生物医学工程系，特拉维夫大学）；Sagol School of Neuroscience, Tel-Aviv University（神经科学学院，特拉维夫大学）
Tokyo University of Agriculture and Technology（东京农工大学）
This paper reports the features of a splashing drop on a solid surface and the temporal evolution, which are extracted through image-sequence classification using a highly interpretable feedforward neural network (FNN) with zero hidden layer. The image sequences used for training-validation and testing of the FNN show the early-stage deformation of milli-sized ethanol drops that impact a hydrophilic glass substrate with the Weber number ranges between 31-474 (splashing threshold about 173). Specific videographing conditions and digital image processing are performed to ensure the high similari…
Hybrid Gaussian Process Regression with Temporal Feature Extraction for Partially Interpretable Remaining Useful Life Interval Prediction in Aeroengine Prognostics
Hyper-STTN: Hypergraph Augmented Spatial-Temporal Transformer Network for Trajectory Prediction
School of Applied and Creative Computing, Purdue University（普渡大学应用与创意计算学院）；Department of Computer Science and Department of Intelligent Systems Engineering, Indiana University Bloomington（印第安纳大学布卢明顿分校计算机科学系和智能系统工程系）；Department of Applied Artificial Intelligence, Sungkyunkwan University（成均馆大学应用人工智能系）
Embedding interpretable $\ell_1$-regression into neural networks for uncovering temporal structure in cell imaging
Institute of Medical Biometry and Statistics (IMBI), Faculty of Medicine and Medical Center, University of Freiburg, Germany（医学生物度量与统计研究所（IMBI）、医学学院与医疗中心、弗赖堡大学）；Institute of Physiology I, Faculty of Medicine, University of Freiburg, Germany（医学生理学研究所I、医学学院、弗赖堡大学）；Department of Computer Science, University of Freiburg, Germany（计算机科学系、弗赖堡大学）；Freiburg Center for Data Analysis, Modeling and AI (FDMAI), University of Freiburg, Germany（弗赖堡数据分析、建模与人工智能中心（FDMAI）、弗赖堡大学）；Centre for Integrative Biological Signalling Studies (CIBSS), University of Freiburg, Germany（整合生物信号研究中心（CIBSS）、弗赖堡大学）
Georgia Institute of Technology（佐治亚理工学院）；KAUST（阿卜杜拉国王科技大学）
Performing driving behaviors based on causal reasoning is essential to ensure driving safety. In this work, we investigated how state-of-the-art 3D Convolutional Neural Networks (CNNs) perform on classifying driving behaviors based on causal reasoning. We proposed a perturbation-based visual explanation method to inspect the models' performance visually. By examining the video attention saliency, we found that existing models could not precisely capture the causes (e.g., traffic light) of the specific action (e.g., stopping). Therefore, the Temporal Reasoning Block (TRB) was proposed and intro…
