Predicting the future trajectory of surrounding vehicles is essential for the navigation of autonomous vehicles in complex real-world driving scenarios. It is challenging as a vehicle's motion is affected by many factors, including its surrounding infrastructures and vehicles. In this work, we develop the ReCoG (Recurrent Convolutional and Graph Neural Networks), which is a general scheme that represents vehicle interactions with infrastructure information as a heterogeneous graph and applies graph neural networks (GNNs) to model the high-level interactions for trajectory prediction. Nodes in the graph contain corresponding features, where a vehicle node contains its sequential feature encoded using Recurrent Neural Network (RNN), and an infrastructure node contains spatial feature encoded using Convolutional Neural Network (CNN). Then the ReCoG predicts the future trajectory of the target vehicle by jointly considering all of the features. Experiments are conducted by using the INTERACTION dataset. Experimental results show that the proposed ReCoG outperforms other state-of-the-art methods in terms of different types of displacement error, validating the feasibility and effectiveness of the developed approach.
To drive safely in complex traffic environments, autonomous vehicles need to make an accurate prediction of the future trajectories of nearby heterogeneous traffic agents (i.e., vehicles, pedestrians, bicyclists, etc). Due to the interactive nature, human drivers are accustomed to infer what the future situations will become if they are going to execute different maneuvers. To fully exploit the impacts of interactions, this paper proposes a ego-planning guided multi-graph convolutional network (EPG-MGCN) to predict the trajectories of heterogeneous agents using both historical trajectory infor…
Reasoning about human motion is an important prerequisite to safe and socially-aware robotic navigation. As a result, multi-agent behavior prediction has become a core component of modern human-robot interactive systems, such as self-driving cars. While there exist many methods for trajectory forecasting, most do not enforce dynamic constraints and do not account for environmental information (e.g., maps). Towards this end, we present Trajectron++, a modular, graph-structured recurrent model that forecasts the trajectories of a general number of diverse agents while incorporating agent dynamic…
Skolkovo Institute of Science and Technology（斯科尔科沃科学技术研究院）
We propose a novel human-swarm interaction system, allowing the user to directly control a swarm of drones in a complex environment through trajectory drawing with a hand gesture interface based on the DNN-based gesture recognition. The developed CV-based system allows the user to control the swarm behavior without additional devices through human gestures and motions in real-time, providing convenient tools to change the swarm's shape and formation. The two types of interaction were proposed and implemented to adjust the swarm hierarchy: trajectory drawing and free-form trajectory generation …
Baidu Research（百度研究院）；Baidu Inc（百度公司）；The University of Hong Kong（香港大学）；The Chinese University of Hong Kong（香港中文大学）；University of Maryland at College Park（马里兰大学帕克分校）
To safely and efficiently navigate in complex urban traffic, autonomous vehicles must make responsible predictions in relation to surrounding traffic-agents (vehicles, bicycles, pedestrians, etc.). A challenging and critical task is to explore the movement patterns of different traffic-agents and predict their future trajectories accurately to help the autonomous vehicle make reasonable navigation decision. To solve this problem, we propose a long short-term memory-based (LSTM-based) realtime traffic prediction algorithm, TrafficPredict. Our approach uses an instance layer to learn instances' …
Heterogeneous Graph-based Trajectory Prediction using Local Map Context and Social Interactions
FZI Research Center for Information Technology（FZI信息技术研究中心）；Karlsruhe Institute of Technology（卡尔斯鲁厄理工学院）；Bosch Center for Artificial Intelligence（博世人工智能中心）
In autonomous driving, perceiving the driving behaviors of surrounding agents is important for the ego-vehicle to make a reasonable decision. In this paper, we propose a neural network model based on trajectories information for driving behavior recognition. Unlike existing trajectory-based methods that recognize the driving behavior using the hand-crafted features or directly encoding the trajectory, our model involves a Multi-Scale Convolutional Neural Network (MSCNN) module to automatically extract the high-level features which are supposed to encode the rich spatial and temporal informatio…
Trajectory prediction for heterogeneous agents: A performance analysis on small and imbalanced datasets
Robert Bosch GmbH, Corporate Research（罗伯特·博世有限公司企业研究部）；Finnish Center for Artificial Intelligence (FCAI)（芬兰人工智能中心）；Technical University of Munich（慕尼黑工业大学）
University of Maryland（马里兰大学）；University of North Carolina（北卡罗来纳大学）
We present a new algorithm for predicting the near-term trajectories of road-agents in dense traffic videos. Our approach is designed for heterogeneous traffic, where the road-agents may correspond to buses, cars, scooters, bicycles, or pedestrians. We model the interactions between different road-agents using a novel LSTM-CNN hybrid network for trajectory prediction. In particular, we take into account heterogeneous interactions that implicitly accounts for the varying shapes, dynamics, and behaviors of different road agents. In addition, we model horizon-based interactions which are used to …
Nanyang Technological University（南洋理工大学）；University of Oxford（牛津大学）
Simultaneous trajectory prediction for multiple heterogeneous traffic participants is essential for the safe and efficient operation of connected automated vehicles under complex driving situations in the real world. The multi-agent prediction task is challenging, as the motions of traffic participants are affected by many factors, including their individual dynamics, their interactions with surrounding agents, the traffic infrastructures, and the number and modalities of the target agents. To further advance the trajectory prediction techniques, in this work we propose a three-channel framewo…
University of Maryland, College Park（马里兰大学帕克分校）；University of North Carolina, Chapel Hill（北卡罗来纳大学教堂山分校）
We present RobustTP, an end-to-end algorithm for predicting future trajectories of road-agents in dense traffic with noisy sensor input trajectories obtained from RGB cameras (either static or moving) through a tracking algorithm. In this case, we consider noise as the deviation from the ground truth trajectory. The amount of noise depends on the accuracy of the tracking algorithm. Our approach is designed for dense heterogeneous traffic, where the road agents corresponding to a mixture of buses, cars, scooters, bicycles, or pedestrians. RobustTP is an approach that first computes trajectories…
University of California, Berkeley（加州大学伯克利分校）；Carnegie Mellon University（卡内基梅隆大学）
Human robot collaboration (HRC) is becoming increasingly important as the paradigm of manufacturing is shifting from mass production to mass customization. The introduction of HRC can significantly improve the flexibility and intelligence of automation. However, due to the stochastic and time-varying nature of human collaborators, it is challenging for the robot to efficiently and accurately identify the plan of human and respond in a safe manner. To address this challenge, we propose an integrated human robot collaboration framework in this paper which includes both plan recognition and traje…
Learning from demonstration (LfD) and imitation learning offer new paradigms for transferring task behavior to robots. A class of methods that enable such online learning require the robot to observe the task being performed and decompose the sensed streaming data into sequences of state-action pairs, which are then input to the methods. Thus, recognizing the state-action pairs correctly and quickly in sensed data is a crucial prerequisite for these methods. We present SA-Net a deep neural network architecture that recognizes state-action pairs from RGB-D data streams. SA-Net performed well in…
Toyota Research Institute（丰田研究院）；NVIDIA Research（英伟达研究院）；Stanford University（斯坦福大学）
Reasoning about the future behavior of other agents is critical to safe robot navigation. The multiplicity of plausible futures is further amplified by the uncertainty inherent to agent state estimation from data, including positions, velocities, and semantic class. Forecasting methods, however, typically neglect class uncertainty, conditioning instead only on the agent's most likely class, even though perception models often return full class distributions. To exploit this information, we present HAICU, a method for heterogeneous-agent trajectory forecasting that explicitly incorporates agent…
We present a reward-predictive, model-based deep learning method featuring trajectory-constrained visual attention for local planning in visual navigation tasks. Our method learns to place visual attention at locations in latent image space which follow trajectories caused by vehicle control actions to enhance predictive accuracy during planning. The attention model is jointly optimized by the task-specific loss and an additional trajectory-constraint loss, allowing adaptability yet encouraging a regularized structure for improved generalization and reliability. Importantly, visual attention i…
Robust and Modular Multi-Limb Synchronization in Motion Stack for Space Robots with Trajectory Clamping via Hypersphere
Space Robotics Lab. (SRL) in Department of Aerospace Engineering, Graduate School of Engineering, Tohoku University（太空机器人实验室（SRL）位于东京东京大学工学研究生院航空航天工程系）
Cooperative Risk-Aware Exploration in Heterogeneous Multi-Robot Systems Using Algorithmic Altruism
School of Electrical and Computer Engineering at Oklahoma State University（俄克拉荷马州立大学电气与计算机工程学院）；Instituto Tecnológico de Aeronáutica（航空技术研究所）；University of California, Santa Barbara（加州大学圣巴巴拉分校）；University of North Carolina, Chapel Hill（北卡罗来纳大学教堂山分校）
The University of Texas at Austin（德克萨斯大学奥斯汀分校）；UT Southwestern Medical Center（德克萨斯大学西南医学中心）
Surgical activity recognition and prediction can help provide important context in many Robot-Assisted Surgery (RAS) applications, for example, surgical progress monitoring and estimation, surgical skill evaluation, and shared control strategies during teleoperation. Transformer models were first developed for Natural Language Processing (NLP) to model word sequences and soon the method gained popularity for general sequence modeling tasks. In this paper, we propose the novel use of a Transformer model for three tasks: gesture recognition, gesture prediction, and trajectory prediction during R…
Institute for Management Cybernetics, Aachen, Germany（亚琛管理控制论研究所）
This paper aims to explore the problem of trajectory prediction in heterogeneous pedestrian zones, where social dynamics representation is a big challenge. Proposed is an end-to-end learning framework for prediction accuracy improvement based on an attention mechanism to learn social interaction from multi-factor inputs.
