Pedestrian behavior prediction is one of the major challenges for intelligent driving systems. Pedestrians often exhibit complex behaviors influenced by various contextual elements. To address this problem, we propose BiPed, a multitask learning framework that simultaneously predicts trajectories and actions of pedestrians by relying on multimodal data. Our method benefits from 1) a bifold encoding approach where different data modalities are processed independently allowing them to develop their own representations, and jointly to produce a representation for all modalities using shared parameters; 2) a novel interaction modeling technique that relies on categorical semantic parsing of the scenes to capture interactions between target pedestrians and their surroundings; and 3) a bifold prediction mechanism that uses both independent and shared decoding of multimodal representations. Using public pedestrian behavior benchmark datasets for driving, PIE and JAAD, we highlight the benefits of the proposed method for behavior prediction and show that our model achieves state-of-the-art performance and improves trajectory and action prediction by up to 22% and 9% respectively. We further investigate the contributions of the proposed reasoning techniques via extensive ablation studies.
Stanford University（斯坦福大学）；Toyota Research Institute（丰田研究院）；Department of Computer Science, Stanford University（斯坦福大学计算机科学系）
Reasoning over visual data is a desirable capability for robotics and vision-based applications. Such reasoning enables forecasting of the next events or actions in videos. In recent years, various models have been developed based on convolution operations for prediction or forecasting, but they lack the ability to reason over spatiotemporal data and infer the relationships of different objects in the scene. In this paper, we present a framework based on graph convolution to uncover the spatiotemporal relationships in the scene for reasoning about pedestrian intent. A scene graph is built on t…
International Institute for Information Technology, Hyderabad（海得拉巴国际信息技术学院）
Multi-view Detection (MVD) is highly effective for occlusion reasoning in a crowded environment. While recent works using deep learning have made significant advances in the field, they have overlooked the generalization aspect, which makes them impractical for real-world deployment. The key novelty of our work is to formalize three critical forms of generalization and propose experiments to evaluate them: generalization with i) a varying number of cameras, ii) varying camera positions, and finally, iii) to new scenes. We find that existing state-of-the-art models show poor generalization by o…
Northwestern Polytechnical University（西北工业大学）；The University of Sydney（悉尼大学）；SenseTime（商汤科技）
Pedestrian counting is a fundamental tool for understanding pedestrian patterns and crowd flow analysis. Existing works (e.g., image-level pedestrian counting, crossline crowd counting et al.) either only focus on the image-level counting or are constrained to the manual annotation of lines. In this work, we propose to conduct the pedestrian counting from a new perspective - Video Individual Counting (VIC), which counts the total number of individual pedestrians in the given video (a person is only counted once). Instead of relying on the Multiple Object Tracking (MOT) techniques, we propose t…
MUSON: A Reasoning-oriented Multimodal Dataset for Socially Compliant Navigation in Urban Environments
Graduate School of Information Science and Technology, Hokkaido University（北海道大学信息科学研究生院）；Graduate School of Computer Science, George Mason University（乔治·马歇尔大学计算机科学研究生院）
Tata Consultancy Services（塔塔咨询服务公司）；Indian Institute of Technology Guwahati（古瓦哈提印度理工学院）
This paper looks into the problem of pedestrian tracking using a monocular, potentially moving, uncalibrated camera. The pedestrians are located in each frame using a standard human detector, which are then tracked in subsequent frames. This is a challenging problem as one has to deal with complex situations like changing background, partial or full occlusion and camera motion. In order to carry out successful tracking, it is necessary to resolve associations between the detected windows in the current frame with those obtained from the previous frame. Compared to methods that use temporal win…
SA-Person: Text-Based Person Retrieval with Scene-aware Re-ranking
School of Computer Science and Technology, Soochow University（苏州大学计算机科学与技术学院）；Centre for Artificial Intelligence and Robotics, Hong Kong Institute of Science and Innovation, Chinese Academy of Sciences（中国科学院香港创新研究院人工智能与机器人中心）；Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统（MAIS））；Wuhan University（武汉大学）；University of Chinese Academy of Sciences（中国科学院大学）
Department of Electrical Engineering and Computer Science, York University（约克大学电气工程与计算机科学系）；York University（约克大学）；Department of Electrical Engineering and Computer Science（电气工程与计算机科学系）；Toronto（多伦多）；ON, Canada（加拿大安大略省）
One of the major challenges that autonomous cars are facing today is driving in urban environments. To make it a reality, autonomous vehicles require the ability to communicate with other road users and understand their intentions. Such interactions are essential between the vehicles and pedestrians as the most vulnerable road users. Understanding pedestrian behavior, however, is not intuitive and depends on various factors such as demographics of the pedestrians, traffic dynamics, environmental conditions, etc. In this paper, we identify these factors by surveying pedestrian behavior studies,…
Unified Vision-Centric Pedestrian Crossing Action Prediction via Adaptive Patch Projection and Proactive Spatial Rectification
School of Electronics and Control Engineering, Chang’an University（长安大学电子与控制工程学院）；School of Astronautics, Northwestern Polytechnical University（西北工业大学航天学院）
Is VLA Reasoning Faithful? Probing Safety of Chain-of-Causation in Autonomous Driving Models
School of Computer Science and Engineering（计算机科学与工程学院）；Central South University（中南大学）；School of Computer Science（计算机科学学院）；University of Wollongong in Dubai（迪拜大学）
Three-Step Hierarchical Transformer for Multi-Pedestrian Trajectory Prediction
IMT Nord Europe, University of Lille, CNRS UMR 9189 - CRIStAL（IMT 北欧洲、里尔大学、法国国家科学研究中心 UMR 9189 - CRIStAL）；University of Lille, CNRS UMR 9189 - CRIStAL（里尔大学、法国国家科学研究中心 UMR 9189 - CRIStAL）；Explain
Inception Institute of Artificial Intelligence（起源人工智能研究院）；University of Verona（维罗纳大学）；Osram GmbH（欧司朗有限公司）；Italian Institute of Technology（意大利理工学院）；Ulm University（乌尔姆大学）
In this work, we explore the correlation between people trajectories and their head orientations. We argue that people trajectory and head pose forecasting can be modelled as a joint problem. Recent approaches on trajectory forecasting leverage short-term trajectories (aka tracklets) of pedestrians to predict their future paths. In addition, sociological cues, such as expected destination or pedestrian interaction, are often combined with tracklets. In this paper, we propose MiXing-LSTM (MX-LSTM) to capture the interplay between positions and head orientations (vislets) thanks to a joint uncon…
Honda Research Institute, USA（美国本田研究院）；Sungkyunkwan University（成均馆大学）
We propose a Deep RObust Goal-Oriented trajectory prediction Network (DROGON) for accurate vehicle trajectory prediction by considering behavioral intentions of vehicles in traffic scenes. Our main insight is that the behavior (i.e., motion) of drivers can be reasoned from their high level possible goals (i.e., intention) on the road. To succeed in such behavior reasoning, we build a conditional prediction model to forecast goal-oriented trajectories with the following stages: (i) relational inference where we encode relational interactions of vehicles using the perceptual context; (ii) intent…
Pedestrian Intention Prediction via Vision-Language Foundation Models
Institute for Transport Studies, Faculty of Environment, Computer Vision and Machine Learning Group, University of Leeds（运输研究学院、环境学院、计算机视觉与机器学习小组、莱斯特大学）；AI Centre, Department of Computer Science, University College London（人工智能中心、计算机科学系、伦敦大学学院）
FaLCon: Facet-Anchored Retrieval with Late Consensus for Sim2Real Text-Based Person Anomaly Search
Vietnamese-German University（越南-德国大学）；Ho Chi Minh City University of Technology（胡志明市技术大学）；University of Information Technology（信息技术大学）；Ho Chi Minh city University of Science（胡志明市科学大学）
Heterogeneous Vision-Language Ensemble with Disagreement-Aware Reranking for Text-Based Person Anomaly Retrieval
Hanoi University of Science and Technology（河内科技大学）；University of Information Technology, VNU-HCM（胡志明市国家大学信息技术大学）；Vietnam National University, Ho Chi Minh City（胡志明市国家大学）；VinUniversity；Vietnamese-German University（越南德国大学）
Progressive Pretext Task Learning for Human Trajectory Prediction
School of Computer Science and Engineering, Sun Yat-sen University（中山大学计算机科学与工程学院）；Guangdong Province Key Laboratory of Information Security Technology（广东省信息安全技术重点实验室）；Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education（教育部机器智能与先进计算重点实验室）
Honda Research Institute USA（本田美国研究所）；University of California, Berkeley（加州大学伯克利分校）；Honda R&D Co., Ltd.（本田技术研究所）
Recent advances in trajectory prediction have shown that explicit reasoning about agents' intent is important to accurately forecast their motion. However, the current research activities are not directly applicable to intelligent and safety critical systems. This is mainly because very few public datasets are available, and they only consider pedestrian-specific intents for a short temporal horizon from a restricted egocentric view. To this end, we propose LOKI (LOng term and Key Intentions), a novel large-scale dataset that is designed to tackle joint trajectory and intention prediction for …
Shanghai Jiao Tong University（上海交通大学）；National University of Singapore（新加坡国立大学）；Shanghai AI Laboratory（上海人工智能实验室）
Learning to predict agent motions with relationship reasoning is important for many applications. In motion prediction tasks, maintaining motion equivariance under Euclidean geometric transformations and invariance of agent interaction is a critical and fundamental principle. However, such equivariance and invariance properties are overlooked by most existing methods. To fill this gap, we propose EqMotion, an efficient equivariant motion prediction model with invariant interaction reasoning. To achieve motion equivariance, we propose an equivariant geometric feature learning module to learn a …
Accurate prediction of others' trajectories is essential for autonomous driving. Trajectory prediction is challenging because it requires reasoning about agents' past movements, social interactions among varying numbers and kinds of agents, constraints from the scene context, and the stochasticity of human behavior. Our approach models these interactions and constraints jointly within a novel Multi-Agent Tensor Fusion (MATF) network. Specifically, the model encodes multiple agents' past trajectories and the scene context into a Multi-Agent Tensor, then applies convolutional fusion to capture m…
