According to several reports published by worldwide organisations, thousands of pedestrians die in road accidents every year. Due to this fact, vehicular technologies have been evolving with the intent of reducing these fatalities. This evolution has not finished yet since, for instance, the predictions of pedestrian paths could improve the current Automatic Emergency Braking Systems (AEBS). For this reason, this paper proposes a method to predict future pedestrian paths, poses and intentions up to 1s in advance. This method is based on Balanced Gaussian Process Dynamical Models (B-GPDMs), which reduce the 3D time-related information extracted from keypoints or joints placed along pedestrian bodies into low-dimensional spaces. The B-GPDM is also capable of inferring future latent positions and reconstruct their associated observations. However, learning a generic model for all kind of pedestrian activities normally provides less ccurate predictions. For this reason, the proposed method obtains multiple models of four types of activity, i.e. walking, stopping, starting and standing, and selects the most similar model to estimate future pedestrian states. This method detects starting activities 125ms after the gait initiation with an accuracy of 80% and recognises stopping intentions 58.33ms before the event with an accuracy of 70%. Concerning the path prediction, the mean error for stopping activities at a Time-To-Event (TTE) of 1s is 238.01mm and, for starting actions, the mean error at a TTE of 0s is 331.93mm.
In order to be globally deployed, autonomous cars must guarantee the safety of pedestrians. This is the reason why forecasting pedestrians' intentions sufficiently in advance is one of the most critical and challenging tasks for autonomous vehicles. This work tries to solve this problem by jointly predicting the intention and visual states of pedestrians. In terms of visual states, whereas previous work focused on x-y coordinates, we will also predict the size and indeed the whole bounding box of the pedestrian. The method is a recurrent neural network in a multi-task learning approach. It has…
TrajFusionNet+: Transformer-Based Prediction of Pedestrian Crossing Intention via Fusion of Trajectory Representations and Scene Graphs
Perception, Robotics and Intelligent Machines Research Group (PRIME)（感知、机器人与智能机器研究组（PRIME））；Department of Computer Science, Université de Moncton（蒙克顿大学计算机科学系）
Intention Enhanced Diffusion Model for Multimodal Pedestrian Trajectory Prediction
Guangdong Provincial Key Laboratory of Fully Actuated System Control Theory and Technology, the Southern University of Science and Technology, Shenzhen 518055, China（广东省全自动化系统控制理论与技术重点实验室，南方科技大学，深圳518055，中国）；Department of Mechanical Engineering, City University of Hong Kong, Hong Kong SAR, China（香港城市大学机械工程系，香港特别行政区，中国）
School of Information Technology, Halmstad University（哈姆斯塔德大学信息技术学院）；RISE Viktoria
Action and intention recognition of pedestrians in urban settings are challenging problems for Advanced Driver Assistance Systems as well as future autonomous vehicles to maintain smooth and safe traffic. This work investigates a number of feature extraction methods in combination with several machine learning algorithms to build knowledge on how to automatically detect the action and intention of pedestrians in urban traffic. We focus on the motion and head orientation to predict whether the pedestrian is about to cross the street or not. The work is based on the Joint Attention for Autonomou…
Predicting vulnerable road user behavior is an essential prerequisite for deploying Automated Driving Systems (ADS) in the real-world. Pedestrian crossing intention should be recognized in real-time, especially for urban driving. Recent works have shown the potential of using vision-based deep neural network models for this task. However, these models are not robust and certain issues still need to be resolved. First, the global spatio-temproal context that accounts for the interaction between the target pedestrian and the scene has not been properly utilized. Second, the optimum strategy for …
Minimizing traffic accidents between vehicles and pedestrians is one of the primary research goals in intelligent transportation systems. To achieve the goal, pedestrian orientation recognition and prediction of pedestrian's crossing or not-crossing intention play a central role. Contemporary approaches do not guarantee satisfactory performance due to limited field-of-view, lack of generalization, and high computational complexity. To overcome these limitations, we propose a real-time predictive pedestrian collision warning service (P2CWS) for two tasks: pedestrian orientation recognition (100…
University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Trajectory prediction is one of the key capabilities for robots to safely navigate and interact with pedestrians. Critical insights from human intention and behavioral patterns need to be integrated to effectively forecast long-term pedestrian behavior. Thus, we propose a framework incorporating a Mutable Intention Filter and a Warp LSTM (MIF-WLSTM) to simultaneously estimate human intention and perform trajectory prediction. The Mutable Intention Filter is inspired by particle filtering and genetic algorithms, where particles represent intention hypotheses that can be mutated throughout the p…
GTransPDM: A Graph-embedded Transformer with Positional Decoupling for Pedestrian Crossing Intention Prediction
Department of Traffic Information and Control Engineering, Jilin University（交通信息与控制工程系，吉林大学）；BIT-Barcelona Innovative Transportation Research Group, Civil Engineering School, UPC Barcelona Tech（巴塞罗那创新交通研究组，土木工程学院，UPC巴塞罗那技术学院）；Computer Vision Center (CVC), Computer Science Department, Universitat Autònoma de Barcelona (UAB)（计算机视觉中心（CVC），计算机科学系，巴塞罗那自治大学（UAB））
针对行人位置畸变及多模态时序建模难题，本文提出 GTransPDM，通过位置解耦与图嵌入 Transformer 融合姿态、位置和车辆运动特征，在 PIE 和 JAAD 上取得最优性能。
TrajFusionNet: Pedestrian Crossing Intention Prediction via Fusion of Sequential and Visual Trajectory Representations
Perception, Robotics and Intelligent Machines Research Group (PRIME) Department of Computer Science, Université de Moncton, Canada（感知、机器人与智能机器研究组（PRIME）计算机科学系，新不伦瑞克大学）
The pedestrian crossing intention prediction problem is to estimate whether or not the target pedestrian will cross the street. State-of-the-art techniques heavily depend on visual data acquired through the front camera of the ego-vehicle to make a prediction of the pedestrian's crossing intention. Hence, the efficiency of current methodologies tends to decrease notably in situations where visual input is imprecise, for instance, when the distance between the pedestrian and ego-vehicle is considerable or the illumination levels are inadequate. To address the limitation, in this paper, we prese…
Universitat Autònoma de Barcelona（巴塞罗那自治大学）；Computer Vision Center（计算机视觉中心）
Anticipating the intentions of vulnerable road users (VRUs) such as pedestrians and cyclists is critical for performing safe and comfortable driving maneuvers. This is the case for human driving and, thus, should be taken into account by systems providing any level of driving assistance, from advanced driver assistant systems (ADAS) to fully autonomous vehicles (AVs). In this paper, we show how the latest advances on monocular vision-based human pose estimation, i.e. those relying on deep Convolutional Neural Networks (CNNs), enable to recognize the intentions of such VRUs. In the case of cycl…
VIT-Ped: Visionary Intention Transformer for Pedestrian Behavior Analysis
German University in Cairo（埃及德国大学）；Faculty of Media Engineering and Technology（媒体工程与技术学院）；Computer Science and Engineering Department（计算机科学与工程系）；C-DRiVeS Lab（C-DRiVeS实验室）；Cognitive Driving Research in Vehicular Systems（车载系统认知驾驶研究）
Understanding pedestrian crossing behavior is an essential goal in intelligent vehicle development, leading to an improvement in their security and traffic flow. In this paper, we developed a method called IntFormer. It is based on transformer architecture and a novel convolutional video classification model called RubiksNet. Following the evaluation procedure in a recent benchmark, we show that our model reaches state-of-the-art results with good performance ($\approx 40$ seq. per second) and size ($8\times $smaller than the best performing model), making it suitable for real-time usage. We a…
Chalmers University of Technology（查尔姆斯理工大学）；Volvo Cars Technology USA（沃尔沃汽车美国技术公司）；Volvo Car Corporation（沃尔沃汽车公司）；University of California, Berkeley（加州大学伯克利分校）
Pedestrian intention recognition is very important to develop robust and safe autonomous driving (AD) and advanced driver assistance systems (ADAS) functionalities for urban driving. In this work, we develop an end-to-end pedestrian intention framework that performs well on day- and night- time scenarios. Our framework relies on objection detection bounding boxes combined with skeletal features of human pose. We study early, late, and combined (early and late) fusion mechanisms to exploit the skeletal features and reduce false positives as well to improve the intention prediction performance. …
Predicting pedestrian crossing intention is an indispensable aspect of deploying advanced driving systems (ADS) or advanced driver-assistance systems (ADAS) to real life. State-of-the-art methods in predicting pedestrian crossing intention often rely on multiple streams of information as inputs, each of which requires massive computational resources and heavy network architectures to generate. However, such reliance limits the practical application of the systems. In this paper, driven the the real-world demands of pedestrian crossing intention prediction models with both high efficiency and a…
The University of Sydney（悉尼大学）；SenseTime Computer Vision Research Group（商汤科技计算机视觉研究组）
In crowd scenarios, reliable trajectory prediction of pedestrians requires insightful understanding of their social behaviors. These behaviors have been well investigated by plenty of studies, while it is hard to be fully expressed by hand-craft rules. Recent studies based on LSTM networks have shown great ability to learn social behaviors. However, many of these methods rely on previous neighboring hidden states but ignore the important current intention of the neighbors. In order to address this issue, we propose a data-driven state refinement module for LSTM network (SR-LSTM), which activat…
PIP-Net: Pedestrian Intention Prediction in the Wild
Institute for Transport Studies, Computer Vision and Machine Learning Group, University of Leeds（交通研究所、计算机视觉与机器学习组、利兹大学）；UCL Centre for Artificial Intelligence, Department of Computer Science, University College London（伦敦大学学院人工智能中心、计算机科学系、伦敦大学学院）
Understanding and predicting pedestrian behavior is an important and challenging area of research for realizing safe and effective navigation strategies in automated and advanced driver assistance technologies in urban scenes. This paper focuses on monocular pedestrian action recognition and 3D localization from an egocentric view for the purpose of predicting intention and forecasting future trajectory. A challenge in addressing this problem in urban traffic scenes is attributed to the unpredictable behavior of pedestrians, whereby actions and intentions are constantly in flux and depend on t…
Pedestrian Intention Prediction via Vision-Language Foundation Models
Institute for Transport Studies, Faculty of Environment, Computer Vision and Machine Learning Group, University of Leeds（运输研究学院、环境学院、计算机视觉与机器学习小组、莱斯特大学）；AI Centre, Department of Computer Science, University College London（人工智能中心、计算机科学系、伦敦大学学院）
Smooth handling of pedestrian interactions is a key requirement for Autonomous Vehicles (AV) and Advanced Driver Assistance Systems (ADAS). Such systems call for early and accurate prediction of a pedestrian's crossing/not-crossing behaviour in front of the vehicle. Existing approaches to pedestrian behaviour prediction make use of pedestrian motion, his/her location in a scene and static context variables such as traffic lights, zebra crossings etc. We stress on the necessity of early prediction for smooth operation of such systems. We introduce the influence of vehicle interactions on pedest…
Traffic violation and the flexible and changeable nature of pedestrians make it more difficult to predict pedestrian behavior or intention, which might be a potential safety hazard on the road. Pedestrian motion state (such as walking and standing) directly affects or reflects its intention. In combination with pedestrian motion state and other influencing factors, pedestrian intention can be predicted to avoid unnecessary accidents. In this paper, pedestrian is treated as non-rigid object, which can be represented by a set of two-dimensional key points, and the movement of key point relative …
Understanding the behaviors and intentions of pedestrians is still one of the main challenges for vehicle autonomy, as accurate predictions of their intentions can guarantee their safety and driving comfort of vehicles. In this paper, we address pedestrian crossing prediction in urban traffic environments by linking the dynamics of a pedestrian's skeleton to a binary crossing intention. We introduce TrouSPI-Net: a context-free, lightweight, multi-branch predictor. TrouSPI-Net extracts spatio-temporal features for different time resolutions by encoding pseudo-images sequences of skeletal joints…
Volkswagen Group（大众汽车集团）；Innovation Center California（加州创新中心）
Detecting and predicting the behavior of pedestrians is extremely crucial for self-driving vehicles to plan and interact with them safely. Although there have been several research works in this area, it is important to have fast and memory efficient models such that it can operate in embedded hardware in these autonomous machines. In this work, we propose a novel architecture using spatial-temporal multi-tasking to do camera based pedestrian detection and intention prediction. Our approach significantly reduces the latency by being able to detect and predict all pedestrians' intention in a si…
Stanford University（斯坦福大学）；Toyota Research Institute（丰田研究院）；Department of Computer Science, Stanford University（斯坦福大学计算机科学系）
Reasoning over visual data is a desirable capability for robotics and vision-based applications. Such reasoning enables forecasting of the next events or actions in videos. In recent years, various models have been developed based on convolution operations for prediction or forecasting, but they lack the ability to reason over spatiotemporal data and infer the relationships of different objects in the scene. In this paper, we present a framework based on graph convolution to uncover the spatiotemporal relationships in the scene for reasoning about pedestrian intent. A scene graph is built on t…
University of Applied Sciences Aschaffenburg（阿沙芬堡应用科学大学）；University of Kassel（卡塞尔大学）；Ulm University（乌尔姆大学）
Avoiding collisions with vulnerable road users (VRUs) using sensor-based early recognition of critical situations is one of the manifold opportunities provided by the current development in the field of intelligent vehicles. As especially pedestrians and cyclists are very agile and have a variety of movement options, modeling their behavior in traffic scenes is a challenging task. In this article we propose movement models based on machine learning methods, in particular artificial neural networks, in order to classify the current motion state and to predict the future trajectory of VRUs. Both…
Institute for Intelligent Systems Research and Innovation (IISRI), Deakin University（迪肯大学智能系统研究与创新研究所）
Understanding the behaviors and intentions of humans are one of the main challenges autonomous ground vehicles still faced with. More specifically, when it comes to complex environments such as urban traffic scenes, inferring the intentions and actions of vulnerable road users such as pedestrians become even harder. In this paper, we address the problem of intent action prediction of pedestrians in urban traffic environments using only image sequences from a monocular RGB camera. We propose a real-time framework that can accurately detect, track and predict the intended actions of pedestrians …
Universitat Autònoma de Barcelona（巴塞罗那自治大学）；Computer Vision Center（计算机视觉中心）
Our recent work suggests that, thanks to nowadays powerful CNNs, image-based 2D pose estimation is a promising cue for determining pedestrian intentions such as crossing the road in the path of the ego-vehicle, stopping before entering the road, and starting to walk or bending towards the road. This statement is based on the results obtained on non-naturalistic sequences (Daimler dataset), i.e. in sequences choreographed specifically for performing the study. Fortunately, a new publicly available dataset (JAAD) has appeared recently to allow developing methods for detecting pedestrian intentio…
A low complexity contextual stacked ensemble-learning approach for pedestrian intent prediction
School of Electronic Engineering & Computer Science, Queen Mary University of London（伦敦女王学院电子工程与计算机科学学院）；Department of Engineering, University of Cambridge（剑桥大学工程系）
Accurate prediction of pedestrian crossing behaviors by autonomous vehicles can significantly improve traffic safety. Existing approaches often model pedestrian behaviors using trajectories or poses but do not offer a deeper semantic interpretation of a person's actions or how actions influence a pedestrian's intention to cross in the future. In this work, we follow the neuroscience and psychological literature to define pedestrian crossing behavior as a combination of an unobserved inner will (a probabilistic representation of binary intent of crossing vs. not crossing) and a set of multi-cla…
With the rapid increase in digital technologies, most fields of study include recognition of human activity and intention recognition, which are essential in smart environments. In this study, we equipped the activity recognition system with the ability to recognize intentions by affecting the pace of movement of individuals in the representation of images. Using this technology in various environments such as elevators and automatic doors will lead to identifying those who intend to pass the automatic door from those who are passing by. This system, if applied in elevators and automatic doors…
University of California, San Diego（加利福尼亚大学圣地亚哥分校）；Laboratory for Intelligent and Safe Automobiles（智能与安全汽车实验室）
The intelligent vehicle community has devoted considerable efforts to model driver behavior, and in particular to detect and overcome driver distraction in an effort to reduce accidents caused by driver negligence. However, as the domain increasingly shifts towards autonomous and semi-autonomous solutions, the driver is no longer integral to the decision making process, indicating a need to refocus efforts elsewhere. To this end, we propose to study pedestrian distraction instead. In particular, we focus on detecting pedestrians who are engaged in secondary activities involving their cellphone…
Universidad de Alcalá（阿尔卡拉大学）；Karlsruher Institut für Technologie（卡尔斯鲁厄理工学院）
Pedestrian crossing prediction is a crucial task for autonomous driving. Numerous studies show that an early estimation of the pedestrian's intention can decrease or even avoid a high percentage of accidents. In this paper, different variations of a deep learning system are proposed to attempt to solve this problem. The proposed models are composed of two parts: a CNN-based feature extractor and an RNN module. All the models were trained and tested on the JAAD dataset. The results obtained indicate that the choice of the features extraction method, the inclusion of additional variables such as…
Video-to-Text Pedestrian Monitoring (VTPM): Leveraging Computer Vision and Large Language Models for Privacy-Preserve Pedestrian Activity Monitoring at Intersections
University at Albany, State University of New York（纽约州立大学奥尔巴尼分校）
We present a bootstrapping framework to simultaneously improve multi-person tracking and activity recognition at individual, interaction and social group activity levels. The inference consists of identifying trajectories of all pedestrian actors, individual activities, pairwise interactions, and collective activities, given the observed pedestrian detections. Our method uses a graphical model to represent and solve the joint tracking and recognition problems via multi-stages: (1) activity-aware tracking, (2) joint interaction recognition and occlusion recovery, and (3) collective activity rec…
Video anomaly detection is a core problem in vision. Correctly detecting and identifying anomalous behaviors in pedestrians from video data will enable safety-critical applications such as surveillance, activity monitoring, and human-robot interaction. In this paper, we propose to leverage trajectory localization and prediction for unsupervised pedestrian anomaly event detection. Different than previous reconstruction-based approaches, our proposed framework rely on the prediction errors of normal and abnormal pedestrian trajectories to detect anomalies spatially and temporally. We present exp…
