Post-interactive Multimodal Trajectory Prediction for Autonomous Driving
College of Computer Science and Electronic Engineering, Hunan University（湖南大学计算机与通信工程学院）；College of Mechanical and Vehicle Engineering, Hunan University（湖南大学机械与运载工程学院）；Department of Computer Science, The University of Hong Kong（香港大学计算机科学系）；Department of Civil Engineering, Monash University（莫纳什大学土木工程系）
Ziyi Huang, Yang Li, Dushuai Li, Yao Mu, Hongmao Qin, Nan Zheng
Modeling the interactions among agents for trajectory prediction of autonomous driving has been challenging due to the inherent uncertainty in agents' behavior. The interactions involved in the predicted trajectories of agents, also called post-interactions, have rarely been considered in trajectory prediction models. To this end, we propose a coarse-to-fine Transformer for multimodal trajectory prediction, i.e., Pioformer, which explicitly extracts the post-interaction features to enhance the prediction accuracy. Specifically, we first build a Coarse Trajectory Network to generate coarse trajectories based on the observed trajectories and lane segments, in which the low-order interaction features are extracted with the graph neural networks. Next, we build a hypergraph neural network-based Trajectory Proposal Network to generate trajectory proposals, where the high-order interaction features are learned by the hypergraphs. Finally, the trajectory proposals are sent to the Proposal Refinement Network for further refinement. The observed trajectories and trajectory proposals are concatenated together as the inputs of the Proposal Refinement Network, in which the post-interaction features are learned by combining the previous interaction features and trajectory consistency features. Moreover, we propose a three-stage training scheme to facilitate the learning process. Extensive experiments on the Argoverse 1 dataset demonstrate the superiority of our method. Compared with the baseline HiVT-64, our model has reduced the prediction errors by 4.4%, 8.4%, 14.4%, 5.7% regarding metrics minADE6, minFDE6, MR6, and brier-minFDE6, respectively.
Detecting pedestrians and predicting future trajectories for them are critical tasks for numerous applications, such as autonomous driving. Previous methods either treat the detection and prediction as separate tasks or simply add a trajectory regression head on top of a detector. In this work, we present a novel end-to-end two-stage network: Spatio-Temporal-Interactive Network (STINet). In addition to 3D geometry modeling of pedestrians, we model the temporal information for each of the pedestrians. To do so, our method predicts both current and past locations in the first stage, so that each…
Predicting the possible future trajectories of the surrounding dynamic agents is an essential requirement in autonomous driving. These trajectories mainly depend on the surrounding static environment, as well as the past movements of those dynamic agents. Furthermore, the multimodal nature of agent intentions makes the trajectory prediction problem more challenging. All of the existing models consider the target agent as well as the surrounding agents similarly, without considering the variation of physical properties. In this paper, we present a novel deep-learning based framework for multimo…
GoalFlow: Goal-Driven Flow Matching for Multimodal Trajectories Generation in End-to-End Autonomous Driving
School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院）；Horizon Robotics；Nanjing University（南京大学）；Huazhong University of Science & Technology（华中科技大学）；Shanghai AI Laboratory（上海人工智能实验室）
Hanyang University（汉阳大学）；Yonsei University（延世大学）；Carnegie Mellon University（卡内基梅隆大学）；Korea University（高丽大学）；Sogang University（西江大学）；Bosch Research Pittsburgh（博世匹兹堡研究中心）
Multi-agent trajectory forecasting in autonomous driving requires an agent to accurately anticipate the behaviors of the surrounding vehicles and pedestrians, for safe and reliable decision-making. Due to partial observability in these dynamical scenes, directly obtaining the posterior distribution over future agent trajectories remains a challenging problem. In realistic embodied environments, each agent's future trajectories should be both diverse since multiple plausible sequences of actions can be used to reach its intended goals, and admissible since they must obey physical constraints an…
Southern University of Science and Technology（南方科技大学）；University of Birmingham（伯明翰大学）
Pedestrian trajectory forecasting is a fundamental task in multiple utility areas, such as self-driving, autonomous robots, and surveillance systems. The future trajectory forecasting is multi-modal, influenced by physical interaction with scene contexts and intricate social interactions among pedestrians. The mainly existing literature learns representations of social interactions by deep learning networks, while the explicit interaction patterns are not utilized. Different interaction patterns, such as following or collision avoiding, will generate different trends of next movement, thus, th…
GoIRL: Graph-Oriented Inverse Reinforcement Learning for Multimodal Trajectory Prediction
Department of Electronic and Computer Engineering, Hong Kong University of Science and Technology, Hong Kong, China（香港理工大学电子与计算机工程系）；Voyager Research, Didi Chuxing, China（维杰研究，滴滴出行，中国）；Zhuoyu Technology Co., Ltd., Shenzhen, China（珠宇科技有限公司，深圳，中国）
The problem of multimodal intent and trajectory prediction for human-driven vehicles in parking lots is addressed in this paper. Using models designed with CNN and Transformer networks, we extract temporal-spatial and contextual information from trajectory history and local bird's eye view (BEV) semantic images, and generate predictions about intent distribution and future trajectory sequences. Our methods outperform existing models in accuracy, while allowing an arbitrary number of modes, encoding complex multi-agent scenarios, and adapting to different parking maps. To train and evaluate our…
In Argoverse motion forecasting competition, the task is to predict the probabilistic future trajectory distribution for the interested targets in the traffic scene. We use vectorized lane map and 2 s targets' history trajectories as input. Then the model outputs 6 forecasted trajectories with probability for each target.
Multimodal prediction results are essential for trajectory prediction task as there is no single correct answer for the future. Previous frameworks can be divided into three categories: regression, generation and classification frameworks. However, these frameworks have weaknesses in different aspects so that they cannot model the multimodal prediction task comprehensively. In this paper, we present a novel insight along with a brand-new prediction framework by formulating multimodal prediction into three steps: modality clustering, classification and synthesis, and address the shortcomings of…
Pattern Recognition and Human Language Technology Research Center（模式识别与人类语言技术研究中心）；Universitat Politècnica de València（瓦伦西亚理工大学）
Despite the advances achieved by neural models in sequence to sequence learning, exploited in a variety of tasks, they still make errors. In many use cases, these are corrected by a human expert in a posterior revision process. The interactive-predictive framework aims to minimize the human effort spent on this process by considering partial corrections for iteratively refining the hypothesis. In this work, we generalize the interactive-predictive approach, typically applied in to machine translation field, to tackle other multimodal problems namely, image and video captioning. We study the ap…
V-ICAL Bench: Evaluating Video In-Context Learning for Multimodal Agents in Interactive Environments
Shanghai Jiao Tong University（上海交通大学）；South China University of Technology（华南理工大学）；Fudan University（复旦大学）；Microsoft Research Asia（微软亚洲研究院）；The University of Hong Kong（香港大学）；The Chinese University of Hong Kong（香港中文大学）；East China Normal University（华东师范大学）
Multimodal Interactive Lung Lesion Segmentation: A Framework for Annotating PET/CT Images based on Physiological and Anatomical Cues
HIDSS4Health - Helmholtz Information and Data Science School for Health（亥姆霍兹信息与健康数据科学学院（HIDSS4Health））；Institute for AI in Medicine (IKIM), University Hospital Essen（埃森大学医院医学人工智能研究所（IKIM））
TaleForge: Interactive Multimodal System for Personalized Story Creation
1 University of Science, VNU-HCM, Ho Chi Minh City, Vietnam；2 Vietnam National University, Ho Chi Minh City, Vietnam；3 Department of Computer Science \ of Dayton Ohio, United States
In this paper, we present Goal-GAN, an interpretable and end-to-end trainable model for human trajectory prediction. Inspired by human navigation, we model the task of trajectory prediction as an intuitive two-stage process: (i) goal estimation, which predicts the most likely target positions of the agent, followed by a (ii) routing module which estimates a set of plausible trajectories that route towards the estimated goal. We leverage information about the past trajectory and visual context of the scene to estimate a multi-modal probability distribution over the possible goal positions, whic…
InterPruner: Interactive Structured Pruning via Taylor-Implicit Criterion and Language-Prior Modulator for Multimodal Object Detection
Beijing University of Technology（北京工业大学）；Southwest University（西南大学）；China University of Geosciences Wuhan（中国地质大学（武汉））；Tsinghua University（清华大学）；Beijing Institute of Technology（北京理工大学）；Yunnan Normal University（云南师范大学）；Beijing Forestry University（北京林业大学）；Ghent University（根特大学）
Intention Enhanced Diffusion Model for Multimodal Pedestrian Trajectory Prediction
Guangdong Provincial Key Laboratory of Fully Actuated System Control Theory and Technology, the Southern University of Science and Technology, Shenzhen 518055, China（广东省全自动化系统控制理论与技术重点实验室，南方科技大学，深圳518055，中国）；Department of Mechanical Engineering, City University of Hong Kong, Hong Kong SAR, China（香港城市大学机械工程系，香港特别行政区，中国）
ETPDesigner: Multi-Agent Orchestration for Interactive Multimodal Electronic Theater Program
Shanghai University（上海大学）；Shanghai Engineering Research Center of Motion Picture Special Effects（上海电影特效工程技术研究中心）；Institute for Math and AI, Wuhan School of Artificial Intelligence, Wuhan University（武汉大学数学与人工智能研究院武汉人工智能学院）
Understanding and predicting the intention of pedestrians is essential to enable autonomous vehicles and mobile robots to navigate crowds. This problem becomes increasingly complex when we consider the uncertainty and multimodality of pedestrian motion, as well as the implicit interactions between members of a crowd, including any response to a vehicle. Our approach, Probabilistic Crowd GAN, extends recent work in trajectory prediction, combining Recurrent Neural Networks (RNNs) with Mixture Density Networks (MDNs) to output probabilistic multimodal predictions, from which likely modal paths a…
Stanford University（斯坦福大学）；Aibee Inc（Aibee公司）；University of Adelaide（阿德莱德大学）
Predicting the future trajectories of multiple interacting agents in a scene has become an increasingly important problem for many different applications ranging from control of autonomous vehicles and social robots to security and surveillance. This problem is compounded by the presence of social interactions between humans and their physical interactions with the scene. While the existing literature has explored some of these cues, they mainly ignored the multimodal nature of each human's future trajectory. In this paper, we present Social-BiGAT, a graph-based generative adversarial network …
Shanghaitech University（上海科技大学）；Shanghai Engineering Research Center of Intelligent Vision and Imaging（上海智能视觉与成像工程技术研究中心）；School of Information Science and Technology, ShanghaiTech University（上海科技大学信息科学与技术学院）
Generating ``bullet-time'' effects of human free-viewpoint videos is critical for immersive visual effects and VR/AR experience. Recent neural advances still lack the controllable and interactive bullet-time design ability for human free-viewpoint rendering, especially under the real-time, dynamic and general setting for our trajectory-aware task. To fill this gap, in this paper we propose a neural interactive bullet-time generator (iButter) for photo-realistic human free-viewpoint rendering from dense RGB streams, which enables flexible and interactive design for human bullet-time visual effe…
ROMA: Real-time Omni-Multimodal Assistant with Interactive Streaming Understanding
CAS Key Laboratory of AI Safety, Institute of Computing Technology, CAS, Beijing, China（中国科学院人工智能安全重点实验室，计算技术研究所，中国科学院，北京，中国）；University of Chinese Academy of Sciences, Beijing, China（中国科学院大学，北京，中国）；Tsinghua University, Beijing, China（清华大学，北京，中国）
University of Edinburgh（爱丁堡大学）；Monash University（莫纳什大学）
Context plays a significant role in the generation of motion for dynamic agents in interactive environments. This work proposes a modular method that utilises a learned model of the environment for motion prediction. This modularity explicitly allows for unsupervised adaptation of trajectory prediction models to unseen environments and new tasks by relying on unlabelled image data only. We model both the spatial and dynamic aspects of a given environment alongside the per agent motions. This results in more informed motion prediction and allows for performance comparable to the state-of-the-ar…
