We present the pedestrian patterns dataset for autonomous driving. The dataset was collected by repeatedly traversing the same three routes for one week starting at different specific timeslots. The purpose of the dataset is to capture the patterns of social and pedestrian behavior along the traversed routes at different times and to eventually use this information to make predictions about the risk associated with autonomously traveling along different routes. This dataset contains the Full HD videos and GPS data for each traversal. Fast R-CNN pedestrian detection method is applied to the captured videos to count the number of pedestrians at each video frame in order to assess the density of pedestrians along a route. By providing this large-scale dataset to researchers, we hope to accelerate autonomous driving research not only to estimate the risk, both to the public and to the autonomous vehicle but also accelerate research on long-term vision-based localization of mobile robots and autonomous vehicles of the future.
Predicting the behavior of road users, particularly pedestrians, is vital for safe motion planning in the context of autonomous driving systems. Traditionally, pedestrian behavior prediction has been realized in terms of forecasting future trajectories. However, recent evidence suggests that predicting higher-level actions, such as crossing the road, can help improve trajectory forecasting and planning tasks accordingly. There are a number of existing datasets that cater to the development of pedestrian action prediction algorithms, however, they lack certain characteristics, such as bird's ey…
One of the most crucial yet challenging tasks for autonomous vehicles in urban environments is predicting the future behaviour of nearby pedestrians, especially at points of crossing. Predicting behaviour depends on many social and environmental factors, particularly interactions between road users. Capturing such interactions requires a global view of the scene and dynamics of the road users in three-dimensional space. This information, however, is missing from the current pedestrian behaviour benchmark datasets. Motivated by these challenges, we propose 1) a novel graph-based model for predi…
To plan safe trajectories in urban environments, autonomous vehicles must be able to quickly assess the future intentions of dynamic agents. Pedestrians are particularly challenging to model, as their motion patterns are often uncertain and/or unknown a priori. This paper presents a novel changepoint detection and clustering algorithm that, when coupled with offline unsupervised learning of a Gaussian process mixture model (DPGP), enables quick detection of changes in intent and online learning of motion patterns not seen in prior training data. The resulting long-term movement predictions dem…
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Toyota Motor Europe（丰田汽车欧洲公司）；CISPA Helmholtz Center for Information Security（CISPA亥姆霍兹信息安全中心）
Accurate prediction of pedestrian and bicyclist paths is integral to the development of reliable autonomous vehicles in dense urban environments. The interactions between vehicle and pedestrian or bicyclist have a significant impact on the trajectories of traffic participants e.g. stopping or turning to avoid collisions. Although recent datasets and trajectory prediction approaches have fostered the development of autonomous vehicles yet the amount of vehicle-pedestrian (bicyclist) interactions modeled are sparse. In this work, we propose Euro-PVI, a dataset of pedestrian and bicyclist traject…
Pedestrian detection plays an important role in many applications such as autonomous driving. We propose a method that explores semantic segmentation results as self-attention cues to significantly improve the pedestrian detection performance. Specifically, a multi-task network is designed to jointly learn semantic segmentation and pedestrian detection from image datasets with weak box-wise annotations. The semantic segmentation feature maps are concatenated with corresponding convolution features maps to provide more discriminative features for pedestrian detection and pedestrian classificati…
Inception Institute of Artificial Intelligence (IIAI)（人工智能研究院（IIAI））；Aalto University（阿尔托大学）
Pedestrian detection is used in many vision based applications ranging from video surveillance to autonomous driving. Despite achieving high performance, it is still largely unknown how well existing detectors generalize to unseen data. This is important because a practical detector should be ready to use in various scenarios in applications. To this end, we conduct a comprehensive study in this paper, using a general principle of direct cross-dataset evaluation. Through this study, we find that existing state-of-the-art pedestrian detectors, though perform quite well when trained and tested o…
Smooth handling of pedestrian interactions is a key requirement for Autonomous Vehicles (AV) and Advanced Driver Assistance Systems (ADAS). Such systems call for early and accurate prediction of a pedestrian's crossing/not-crossing behaviour in front of the vehicle. Existing approaches to pedestrian behaviour prediction make use of pedestrian motion, his/her location in a scene and static context variables such as traffic lights, zebra crossings etc. We stress on the necessity of early prediction for smooth operation of such systems. We introduce the influence of vehicle interactions on pedest…
Swapping Faces, Saving Features: A Dual-Purpose Pipeline for Pedestrian Privacy in ITS
C-DRiVeS Lab: Cognitive Driving Research in Vehicular Systems（C-DRiVeS实验室：车辆系统中的认知驾驶研究）；Computer Science and Engineering Department - Faculty of Media Engineering and Technology - German University in Cairo（计算机科学与工程系 - 媒体工程与技术学院 - 开罗德国大学）
Stellantis Group（斯泰兰蒂斯集团）；Université de Lorraine（洛林大学）；CNRS（法国国家科学研究中心）；Inria（法国国家信息与自动化研究所）
The human driver is no longer the only one concerned with the complexity of the driving scenarios. Autonomous vehicles (AV) are similarly becoming involved in the process. Nowadays, the development of AVs in urban places raises essential safety concerns for vulnerable road users (VRUs) such as pedestrians. Therefore, to make the roads safer, it is critical to classify and predict the pedestrians' future behavior. In this paper, we present a framework based on multiple variations of the Transformer models able to infer predict the pedestrian street-crossing decision-making based on the dynamics…
Group 42（G42集团）；Inception Institute of Artificial Intelligence（起源人工智能研究院）；The Chinese University of Hong Kong（香港中文大学）；National Center for Artificial Intelligence（国家人工智能中心）
Pedestrian detection is the cornerstone of many vision based applications, starting from object tracking to video surveillance and more recently, autonomous driving. With the rapid development of deep learning in object detection, pedestrian detection has achieved very good performance in traditional single-dataset training and evaluation setting. However, in this study on generalizable pedestrian detectors, we show that, current pedestrian detectors poorly handle even small domain shifts in cross-dataset evaluation. We attribute the limited generalization to two main factors, the method and t…
The Ohio State University（俄亥俄州立大学）；Clausthal University of Technology（克劳斯塔尔工业大学）
In mixed traffic scenarios, a certain number of pedestrians might coexist in a small area while interacting with vehicles. In this situation, every pedestrian must simultaneously react to the surrounding pedestrians and vehicles. Analytical modeling of such collective pedestrian motion can benefit intelligent transportation practices like shared space design and urban autonomous driving. This work proposed the sub-goal social force model (SG-SFM) to describe the collective pedestrian motion under vehicle influence. The proposed model introduced a new design of vehicle influence on pedestrian m…
ARCANE-PedSynth: Synthetic Multi-Pedestrian Datasets with Behavioural Crossing Annotations
Computer Vision Center (CVC), Universitat Aut\` o noma de Barcelona (UAB), Bellaterra, Barcelona, Spain；Institute of Electronics, Faculty of Computer Science, Electronics；Telecommunications, AGH University of Krakow, Krak\' o w, Poland
Multispectral pedestrian detection has attracted increasing attention from the research community due to its crucial competence for many around-the-clock applications (e.g., video surveillance and autonomous driving), especially under insufficient illumination conditions. We create a human baseline over the KAIST dataset and reveal that there is still a large gap between current top detectors and human performance. To narrow this gap, we propose a network fusion architecture, which consists of a multispectral proposal network to generate pedestrian proposals, and a subsequent multispectral cla…
Detecting pedestrians and predicting future trajectories for them are critical tasks for numerous applications, such as autonomous driving. Previous methods either treat the detection and prediction as separate tasks or simply add a trajectory regression head on top of a detector. In this work, we present a novel end-to-end two-stage network: Spatio-Temporal-Interactive Network (STINet). In addition to 3D geometry modeling of pedestrians, we model the temporal information for each of the pedestrians. To do so, our method predicts both current and past locations in the first stage, so that each…
École Polytechnique Fédérale de Lausanne（洛桑联邦理工学院）
Forecasting pedestrians' future motions is essential for autonomous driving systems to safely navigate in urban areas. However, existing prediction algorithms often overly rely on past observed trajectories and tend to fail around abrupt dynamic changes, such as when pedestrians suddenly start or stop walking. We suggest that predicting these highly non-linear transitions should form a core component to improve the robustness of motion prediction algorithms. In this paper, we introduce the new task of pedestrian stop and go forecasting. Considering the lack of suitable existing datasets for it…
Universitat Autonoma de Barcelona（巴塞罗那自治大学）；Computer Vision Centre (CVC)（计算机视觉中心）
We propose a real-time pedestrian detection system for the embedded Nvidia Tegra X1 GPU-CPU hybrid platform. The pipeline is composed by the following state-of-the-art algorithms: Histogram of Local Binary Patterns (LBP) and Histograms of Oriented Gradients (HOG) features extracted from the input image; Pyramidal Sliding Window technique for candidate generation; and Support Vector Machine (SVM) for classification. Results show a 8x speedup in the target Tegra X1 platform and a better performance/watt ratio than desktop CUDA platforms in study.
Changzhou University（常州大学）；University of California, Berkeley（加州大学伯克利分校）；Nanjing Xiaozhuang University（南京晓庄学院）
Pedestrian trajectory prediction in dynamic scenes remains a challenging and critical problem in numerous applications, such as self-driving cars and socially aware robots. Challenges concentrate on capturing pedestrians' motion patterns and social interactions, as well as handling the future uncertainties. Recent studies focus on modeling pedestrians' motion patterns with recurrent neural networks, capturing social interactions with pooling-based or graph-based methods, and handling future uncertainties by using random Gaussian noise as the latent variable. However, they do not integrate spec…
University of Sao Paulo（圣保罗大学）；University of California, San Diego（加州大学圣地亚哥分校）
Pedestrians and vehicles often share the road in complex inner city traffic. This leads to interactions between the vehicle and pedestrians, with each affecting the other's motion. In order to create robust methods to reason about pedestrian behavior and to design interfaces of communication between self-driving cars and pedestrians we need to better understand such interactions. In this paper, we present a data-driven approach to implicitly model pedestrians' interactions with vehicles, to better predict pedestrian behavior. We propose a LSTM model that takes as input the past trajectories of…
Robotics Institute, University of Michigan（密歇根大学机器人研究所）；University of Michigan（密歇根大学）
Pedestrians and drivers interact closely in a wide range of environments. Autonomous vehicles (AVs) correspondingly face the need to predict pedestrians' future trajectories in these same environments. Traditional model-based prediction methods have been limited to making predictions in highly structured scenes with signalized intersections, marked crosswalks, or curbs. Deep learning methods have instead leveraged datasets to learn predictive features that generalize across scenes, at the cost of model interpretability. This paper aims to achieve both widely applicable and interpretable predic…
School of Mechanical Engineering, University of Science and Technology Beijing（北京科技大学机械工程学院）
As a core technology of the autonomous driving system, pedestrian trajectory prediction can significantly enhance the function of active vehicle safety and reduce road traffic injuries. In traffic scenes, when encountering with oncoming people, pedestrians may make sudden turns or stop immediately, which often leads to complicated trajectories. To predict such unpredictable trajectories, we can gain insights into the interaction between pedestrians. In this paper, we present a novel generative method named Spatial Interaction Transformer (SIT), which learns the spatio-temporal correlation of p…
Autonomous Vehicles (AV) will transform transportation, but also the interaction between vehicles and pedestrians. In the absence of a driver, it is not clear how an AV can communicate its intention to pedestrians. One option is to use visual signals. To advance their design, we conduct four human-participant experiments and evaluate six representative AV visual signals for visibility, intuitiveness, persuasiveness, and usability at pedestrian crossings. Based on the results, we distill twelve practical design recommendations for AV visual signals, with focus on signal pattern design and place…
University of Michigan（密歇根大学）；Ford Center for Autonomous Vehicles Laboratory（福特自动驾驶汽车实验室）
This paper presents a novel dataset titled PedX, a large-scale multimodal collection of pedestrians at complex urban intersections. PedX consists of more than 5,000 pairs of high-resolution (12MP) stereo images and LiDAR data along with providing 2D and 3D labels of pedestrians. We also present a novel 3D model fitting algorithm for automatic 3D labeling harnessing constraints across different modalities and novel shape and temporal priors. All annotated 3D pedestrians are localized into the real-world metric space, and the generated 3D models are validated using a mocap system configured in a…
We present the Brown Pedestrian Odometry Dataset (BPOD) for benchmarking visual odometry algorithms in head-mounted pedestrian settings. This dataset was captured using synchronized global and rolling shutter stereo cameras in 12 diverse indoor and outdoor locations on Brown University's campus. Compared to existing datasets, BPOD contains more image blur and self-rotation, which are common in pedestrian odometry but rare elsewhere. Ground-truth trajectories are generated from stick-on markers placed along the pedestrian's path, and the pedestrian's position is documented using a third-person …
Predicting pedestrian motion is essential for developing socially-aware robots that interact in a crowded environment. While the natural visual perspective for a social interaction setting is an egocentric view, the majority of existing work in trajectory prediction therein has been investigated purely in the top-down trajectory space. To support first-person view trajectory prediction research, we present T2FPV, a method for constructing high-fidelity first-person view (FPV) datasets given a real-world, top-down trajectory dataset; we showcase our approach on the ETH/UCY pedestrian dataset to…
National University of Defense Technology（国防科技大学）；NICTA（澳大利亚国家信息与通信技术研究院）；Australian National University（澳大利亚国立大学）
Accurately detecting pedestrians in images plays a critically important role in many computer vision applications. Extraction of effective features is the key to this task. Promising features should be discriminative, robust to various variations and easy to compute. In this work, we present novel features, termed dense center-symmetric local binary patterns (CS-LBP) and pyramid center-symmetric local binary/ternary patterns (CS-LBP/LTP), for pedestrian detection. The standard LBP proposed by Ojala et al. \cite{c4} mainly captures the texture information. The proposed CS-LBP feature, in contra…
UBTECH Sydney Artificial Intelligence Centre, The University of Sydney（悉尼大学优必选悉尼人工智能中心）；The University of Sydney（悉尼大学）
Deep learning-based computer vision is usually data-hungry. Many researchers attempt to augment datasets with synthesized data to improve model robustness. However, the augmentation of popular pedestrian datasets, such as Caltech and Citypersons, can be extremely challenging because real pedestrians are commonly in low quality. Due to the factors like occlusions, blurs, and low-resolution, it is significantly difficult for existing augmentation approaches, which generally synthesize data using 3D engines or generative adversarial networks (GANs), to generate realistic-looking pedestrians. Alte…
The Ohio State University（俄亥俄州立大学）；Dalian University of Technology（大连理工大学）
Predicting the collective motion of a group of pedestrians (a crowd) under the vehicle influence is essential for the development of autonomous vehicles to deal with mixed urban scenarios where interpersonal interaction and vehicle-crowd interaction (VCI) are significant. This usually requires a model that can describe individual pedestrian motion under the influence of nearby pedestrians and the vehicle. This study proposed two pedestrian trajectory datasets, CITR dataset and DUT dataset, so that the pedestrian motion models can be further calibrated and verified, especially when vehicle infl…
Inferring the full transportation network, including sidewalks and cycleways, is crucial for many automated systems, including autonomous driving, multi-modal navigation, trip planning, mobility simulations, and freight management. Many transportation decisions can be informed based on an accurate pedestrian network, its interactions, and connectivity with the road networks of other modes of travel. A connected pedestrian path network is vital to transportation activities, as sidewalks and crossings connect pedestrians to other modes of transportation. However, information about these paths' l…
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
Pedestrian attribute recognition aims to assign multiple attributes to one pedestrian image captured by a video surveillance camera. Although numerous methods are proposed and make tremendous progress, we argue that it is time to step back and analyze the status quo of the area. We review and rethink the recent progress from three perspectives. First, given that there is no explicit and complete definition of pedestrian attribute recognition, we formally define and distinguish pedestrian attribute recognition from other similar tasks. Second, based on the proposed definition, we expose the lim…
Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；Temple University（天普大学）
In this paper, we aim to improve the dataset foundation for pedestrian attribute recognition in real surveillance scenarios. Recognition of human attributes, such as gender, and clothes types, has great prospects in real applications. However, the development of suitable benchmark datasets for attribute recognition remains lagged behind. Existing human attribute datasets are collected from various sources or an integration of pedestrian re-identification datasets. Such heterogeneous collection poses a big challenge on developing high quality fine-grained attribute recognition algorithms. Furth…
Harbin Institute of Technology（哈尔滨工业大学）；Inner Mongolia University（内蒙古大学）
Thermal infrared (TIR) pedestrian tracking is one of the important components among numerous applications of computer vision, which has a major advantage: it can track pedestrians in total darkness. The ability to evaluate the TIR pedestrian tracker fairly, on a benchmark dataset, is significant for the development of this field. However, there is not a benchmark dataset. In this paper, we develop a TIR pedestrian tracking dataset for the TIR pedestrian tracker evaluation. The dataset includes 60 thermal sequences with manual annotations. Each sequence has nine attribute labels for the attribu…
Center for Biometric Security Research（生物特征安全研究中心）；National Laboratory of Pattern Recognition（模式识别国家重点实验室）；Institute of Automation Chinese Academy of Sciences（中国科学院自动化研究所）；University of Chinese Academy of Sciences（中国科学院大学）；Macau University of Science and Technology（澳门科技大学）；University of Southern California（南加利福尼亚大学）；Nanjing University of Aeronautics and Astronautics（南京航空航天大学）；Institute of Deep Learning, Baidu Research（百度研究院深度学习研究所）；National Engineering Laboratory for Deep Learning Technology and Application（深度学习技术及应用国家工程实验室）
Pedestrian detection has achieved significant progress with the availability of existing benchmark datasets. However, there is a gap in the diversity and density between real world requirements and current pedestrian detection benchmarks: 1) most of existing datasets are taken from a vehicle driving through the regular traffic scenario, usually leading to insufficient diversity; 2) crowd scenarios with highly occluded pedestrians are still under represented, resulting in low density. To narrow this gap and facilitate future pedestrian detection research, we introduce a large and diverse datase…
ShanghaiTech University（上海科技大学）；The Chinese University of Hong Kong（香港中文大学）；RWTH Aachen University（亚琛工业大学）；Shanghai AI Laboratory（上海人工智能实验室）；University of Kentucky（肯塔基大学）；University of Maryland at College Park（马里兰大学帕克分校）
Accurately detecting and tracking pedestrians in 3D space is challenging due to large variations in rotations, poses and scales. The situation becomes even worse for dense crowds with severe occlusions. However, existing benchmarks either only provide 2D annotations, or have limited 3D annotations with low-density pedestrian distribution, making it difficult to build a reliable pedestrian perception system especially in crowded scenes. To better evaluate pedestrian perception algorithms in crowded scenarios, we introduce a large-scale multimodal dataset,STCrowd. Specifically, in STCrowd, there…
University of Chinese Academy of Sciences（中国科学院大学）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；CAS Center for Excellence in Brain Science and Intelligence Technology（中国科学院脑科学与智能技术卓越创新中心）
Despite various methods are proposed to make progress in pedestrian attribute recognition, a crucial problem on existing datasets is often neglected, namely, a large number of identical pedestrian identities in train and test set, which is not consistent with practical application. Thus, images of the same pedestrian identity in train set and test set are extremely similar, leading to overestimated performance of state-of-the-art methods on existing datasets. To address this problem, we propose two realistic datasets PETA\textsubscript{$zs$} and RAPv2\textsubscript{$zs$} following zero-shot se…
Vehicles are constantly approaching and sharing the road with pedestrians, and as a result it is critical for vehicles to prevent any collisions with pedestrians. Current methods for pedestrian collision prevention focus on integrating visual pedestrian detectors with Automatic Emergency Braking (AEB) systems which can trigger warnings and apply brakes as a pedestrian enters a vehicle's path. Unfortunately, pedestrian-detection-based systems can be hindered in certain situations such as nighttime or when pedestrians are occluded. Our system, CHAMP (Crowdsourced, History-based Advisories of Map…
Universidad Nacional de San Antonio Abad del Cusco（库斯科圣安东尼奥阿巴德国立大学）；Pontificia Universidad Católica del Perú（秘鲁天主教大学）
The development of autonomous vehicles arises new challenges in urban traffic scenarios where vehicle-pedestrian interactions are frequent e.g. vehicle yields to pedestrians, pedestrian slows down due approaching to the vehicle. Over the last years, several datasets have been developed to model these interactions. However, available datasets do not cover near-accident scenarios that our dataset covers. We introduce I see you, a new vehicle-pedestrian interaction dataset that tackles the lack of trajectory data in near-accident scenarios using YOLOv5 and camera calibration methods. I see you co…
In this work, we contribute an EM algorithm for estimation of corner points and linear crossing segments for both marked and unmarked pedestrian crosswalks using the detections of pedestrians from processed LiDAR point clouds or camera images. We demonstrate the algorithmic performance by analyzing three real-world datasets containing multiple periods of data collection for four-corner and two-corner intersections with marked and unmarked crosswalks. Additionally, we include a Python video tool to visualize the crossing parameter estimation, pedestrian trajectories, and phase intervals in our …
