Dynamic demand prediction is crucial for the efficient operation and management of urban transportation systems. Extensive research has been conducted on single-mode demand prediction, ignoring the fact that the demands for different transportation modes can be correlated with each other. Despite some recent efforts, existing approaches to multimodal demand prediction are generally not flexible enough to account for multiplex networks with diverse spatial units and heterogeneous spatiotemporal correlations across different modes. To tackle these issues, this study proposes a multi-relational spatiotemporal graph neural network (ST-MRGNN) for multimodal demand prediction. Specifically, the spatial dependencies across modes are encoded with multiple intra- and inter-modal relation graphs. A multi-relational graph neural network (MRGNN) is introduced to capture cross-mode heterogeneous spatial dependencies, consisting of generalized graph convolution networks to learn the message passing mechanisms within relation graphs and an attention-based aggregation module to summarize different relations. We further integrate MRGNNs with temporal gated convolution layers to jointly model heterogeneous spatiotemporal correlations. Extensive experiments are conducted using real-world subway and ride-hailing datasets from New York City, and the results verify the improved performance of our proposed approach over existing methods across modes. The improvement is particularly large for demand-sparse locations. Further analysis of the attention mechanisms of ST-MRGNN also demonstrates its good interpretability for understanding cross-mode interactions.
Leveraging Spatiotemporal Graph Neural Networks for Multi-Store Sales Forecasting
Department of Chemical Engineering, IIT Bombay（印度博亚理工大学化学工程系）；Rajiv Gandhi University（拉吉夫·甘地大学）；Tsinghua University（清华大学）；Palmer Research Laboratories（帕勒尔研究实验室）
Kyoto University（京都大学）；RIKEN Center for Advanced Intelligence Project（理化学研究所先进智能项目中心）；NAVER Corporation（NAVER公司）
Multimodal relational data analysis has become of increasing importance in recent years, for exploring across different domains of data, such as images and their text tags obtained from social networking services (e.g., Flickr). A variety of data analysis methods have been developed for visualization; to give an example, t-Stochastic Neighbor Embedding (t-SNE) computes low-dimensional feature vectors so that their similarities keep those of the observed data vectors. However, t-SNE is designed only for a single domain of data but not for multimodal data; this paper aims at visualizing multimod…
ModuLM: Enabling Modular and Multimodal Molecular Relational Learning with Large Language Models
University of Science and Technology of China（中国科学技术大学）；Suzhou Institute for Advanced Research, USTC（先进研究所）；Hunan University（湖南大学）；Monash University（墨尔本大学）；State Key Laboratory of Precision and Intelligent Chemistry, USTC（精密与智能化学国家重点实验室）
Spatiotemporal Multi-Task Graph Transformer for Trip-Level Transit Prediction
Department of Engineering Cybernetics, Norwegian University of Science and Technology (NTNU)（工程 cybernetics 部，挪威科学与技术大学（NTNU））；Department of Computer Science, Norwegian University of Science and Technology (NTNU)（计算机科学部，挪威科学与技术大学（NTNU））
Relational Learning in Pre-Trained Models: A Theory from Hypergraph Recovery Perspective
School of Intelligence Science and Technology, Peking University（北京大学智能科学与技术学院）；Institute for Artificial Intelligence, Peking University（北京大学人工智能研究院）；Pazhou Laboratory (Huangpu)（琶洲实验室（黄埔））；University of Illinois Chicago（芝加哥大学伊利诺伊分校）
Sorbonne Université（索邦大学）；CNRS（法国国家科学研究中心）；Heuritech（赫瑞泰克公司）；Conservatoire National des Arts et Métiers（法国国立工艺学院）
Multimodal attentional networks are currently state-of-the-art models for Visual Question Answering (VQA) tasks involving real images. Although attention allows to focus on the visual content relevant to the question, this simple mechanism is arguably insufficient to model complex reasoning features required for VQA or other high-level tasks. In this paper, we propose MuRel, a multimodal relational network which is learned end-to-end to reason over real images. Our first contribution is the introduction of the MuRel cell, an atomic reasoning primitive representing interactions between question…
Multimodal Fine-grained Reasoning for Post Quality Evaluation
School of Computer Science（计算机科学学院）；School of Artificial Intelligence Institute（人工智能研究所）；State Key Laboratory of Information Security, Institute of Information Engineering（信息安全国家重点实验室，信息工程研究所）；Key Laboratory of Education Informatization for Nationalities（民族教育信息化重点实验室）
Building spatiotemporal activity models for people's activities in urban spaces is important for understanding the ever-increasing complexity of urban dynamics. With the emergence of Geo-Tagged Social Media (GTSM) records, previous studies demonstrate the potential of GTSM records for spatiotemporal activity modeling. State-of-the-art methods for this task embed different modalities (location, time, and text) of GTSM records into a single embedding space. However, they ignore Non-GeoTagged Social Media (NGTSM) records, which generally account for the majority of posts (e.g., more than 95\% in …
Disentangling Bias by Modeling Intra- and Inter-modal Causal Attention for Multimodal Sentiment Analysis
School of Computer Science, South China Normal University（华南师范大学计算机学院）；School of Electronics and Information Technology, Sun Yat-sen University（中山大学电子与信息学院）
Self-supervised representation learning techniques utilize large datasets without semantic annotations to learn meaningful, universal features that can be conveniently transferred to solve a wide variety of downstream supervised tasks. In this work, we propose a self-supervised method for learning representations of geographic locations from unlabeled GPS trajectories to solve downstream geospatial computer vision tasks. Tiles resulting from a raster representation of the earth's surface are modeled as nodes on a graph or pixels of an image. GPS trajectories are modeled as allowed Markovian pa…
Spatiotemporal Heterogeneity of AI-Driven Traffic Flow Patterns and Land Use Interaction: A GeoAI-Based Analysis of Multimodal Urban Mobility
Department of Applied Mathematics（应用数学系）；Computer Science (DTU Compute), Technical University of Denmark, Kongens Lyngby, Denmark（计算机科学（DTU Compute），丹麦技术大学，Kongens Lyngby）
Spatiotemporal Attention-Augmented Inverse Reinforcement Learning for Multi-Agent Task Allocation
College of Electronic and Information Engineering, Tongji University（同济大学电子与信息工程学院）；Shanghai Research Institute for Intelligent Autonomous Systems, Tongji University（同济大学智能自主系统上海研究院）；Graz University of Technology（格拉茨技术大学）；Virtual Vehicle Research GmbH（虚拟车辆研究公司）
End-to-end multimodal learning on knowledge graphs has been left largely unaddressed. Instead, most end-to-end models such as message passing networks learn solely from the relational information encoded in graphs' structure: raw values, or literals, are either omitted completely or are stripped from their values and treated as regular nodes. In either case we lose potentially relevant information which could have otherwise been exploited by our learning methods. To avoid this, we must treat literals and non-literals as separate cases. We must also address each modality separately and accordin…
Stock Movement Prediction with Multimodal Stable Fusion via Gated Cross-Attention Mechanism
School of Information and Electronic Engineering, Zhejiang University of Science and Technology（浙江理工大学信息与电子工程学院）；Department of Finance Accounting and Economics, Business School of Nottingham University（诺丁汉大学商学院金融会计与经济学系）
Resource-Efficient Beam Prediction in mmWave Communications with Multimodal Realistic Simulation Framework
Department of Computer Science and Engineering, Kyung Hee University（计算机科学与工程系，庆尚大学）；Bradley Department of Electrical and Computer Engineering, Virginia Tech（电气与计算机工程系，弗吉尼亚理工大学）；Department of Electronic Systems, Aalborg University（电子系统系，奥尔堡大学）
Developing safe human-robot interaction systems is a necessary step towards the widespread integration of autonomous agents in society. A key component of such systems is the ability to reason about the many potential futures (e.g. trajectories) of other agents in the scene. Towards this end, we present the Trajectron, a graph-structured model that predicts many potential future trajectories of multiple agents simultaneously in both highly dynamic and multimodal scenarios (i.e. where the number of agents in the scene is time-varying and there are many possible highly-distinct futures for each …
Carnegie Mellon University（卡内基梅隆大学）；Singapore University of Technology and Design（新加坡科技设计大学）
Human communication is multimodal in nature; it is through multiple modalities such as language, voice, and facial expressions, that opinions and emotions are expressed. Data in this domain exhibits complex multi-relational and temporal interactions. Learning from this data is a fundamentally challenging research problem. In this paper, we propose Modal-Temporal Attention Graph (MTAG). MTAG is an interpretable graph-based neural model that provides a suitable framework for analyzing multimodal sequential data. We first introduce a procedure to convert unaligned multimodal sequence data into a …
UiT The Arctic University of Norway（挪威北极大学）；NORCE (the Norwegian Research Centre)（挪威NORCE研究中心）；University of Genoa（热那亚大学）
Image translation with convolutional neural networks has recently been used as an approach to multimodal change detection. Existing approaches train the networks by exploiting supervised information of the change areas, which, however, is not always available. A main challenge in the unsupervised problem setting is to avoid that change pixels affect the learning of the translation function. We propose two new network architectures trained with loss functions weighted by priors that reduce the impact of change pixels on the learning objective. The change prior is derived in an unsupervised fash…
