Beyond the Individual: Introducing Group Intention Forecasting with SHOT Dataset
National Engineering Research Center for Multimedia Software, Institute of Artificial Intelligence, School of Computer Science（国家多媒体软件工程研究中心、人工智能研究所、计算机科学学院）；Tsinghua University（清华大学）；School of Mathematical Sciences, Peking University（北京大学数学科学学院）；National Engineering Research Center for Multimedia Software, School of Computer Science（国家多媒体软件工程研究中心、计算机科学学院）；School of Computer Science, Peking University（北京大学计算机科学学院）；State Key Laboratory for Multimedia Information Processing（多媒体信息处理国家重点实验室）；School of Computing, National University of Singapore（新加坡国立大学计算机学院）；Hubei Key Laboratory of Transportation Internet of Things, School of Computer Science and Artificial Intelligence（湖北交通物联网重点实验室、计算机科学与人工智能学院）；Wuhan University of Technology（武汉理工大学）
Ruixu Zhang, Yuran Wang, Xinyi Hu, Chaoyu Mai, Wenxuan Liu, Danni Xu, Xian Zhong, Zheng Wang
Intention recognition has traditionally focused on individual intentions, overlooking the complexities of collective intentions in group settings. To address this limitation, we introduce the concept of group intention, which represents shared goals emerging through the actions of multiple individuals, and Group Intention Forecasting (GIF), a novel task that forecasts when group intentions will occur by analyzing individual actions and interactions before the collective goal becomes apparent. To investigate GIF in a specific scenario, we propose SHOT, the first large-scale dataset for GIF, consisting of 1,979 basketball video clips captured from 5 camera views and annotated with 6 types of individual attributes. SHOT is designed with 3 key characteristics: multi-individual information, multi-view adaptability, and multi-level intention, making it well-suited for studying emerging group intentions. Furthermore, we introduce GIFT (Group Intention ForecasTer), a framework that extracts fine-grained individual features and models evolving group dynamics to forecast intention emergence. Experimental results confirm the effectiveness of SHOT and GIFT, establishing a strong foundation for future research in group intention forecasting. The dataset is available at https://xinyi-hu.github.io/SHOT_DATASET.
Technical University of Munich（慕尼黑工业大学）；Google（谷歌）
We propose the task of forecasting characteristic 3d poses: from a short sequence observation of a person, predict a future 3d pose of that person in a likely action-defining, characteristic pose -- for instance, from observing a person picking up an apple, predict the pose of the person eating the apple. Prior work on human motion prediction estimates future poses at fixed time intervals. Although easy to define, this frame-by-frame formulation confounds temporal and intentional aspects of human action. Instead, we define a semantically meaningful pose prediction task that decouples the predi…
Expressive Forecasting of 3D Whole-body Human Motions
MiLAB, Westlake University（西湖大学 MiLAB）；Zhejiang University（浙江大学）；Nanjing University of Science and Technology（南京理工大学）；Shenzhen Graduate School, Peking University（北京大学深圳研究生院）；Xiaohongshu Inc.（小红书公司）
Stanford University（斯坦福大学）；Toyota Research Institute（丰田研究院）；Department of Computer Science, Stanford University（斯坦福大学计算机科学系）
Reasoning over visual data is a desirable capability for robotics and vision-based applications. Such reasoning enables forecasting of the next events or actions in videos. In recent years, various models have been developed based on convolution operations for prediction or forecasting, but they lack the ability to reason over spatiotemporal data and infer the relationships of different objects in the scene. In this paper, we present a framework based on graph convolution to uncover the spatiotemporal relationships in the scene for reasoning about pedestrian intent. A scene graph is built on t…
Intention-Conditioned Long-Term Human Egocentric Action Forecasting
Technische Universität Wien (TU Wien)（维也纳技术大学）；Artificial Intelligence Graduate School (AIGS), Ulsan National Institute of Science and Technology (UNIST)（蔚山科学技术院人工智能研究生院）；Institute of Robotics and Mechatronics, German Aerospace Center（德国航空航天中心机器人与机电一体化研究所）
University of Applied Sciences Aschaffenburg（阿沙芬堡应用科学大学）；University of Kassel（卡塞尔大学）；Ulm University（乌尔姆大学）
Avoiding collisions with vulnerable road users (VRUs) using sensor-based early recognition of critical situations is one of the manifold opportunities provided by the current development in the field of intelligent vehicles. As especially pedestrians and cyclists are very agile and have a variety of movement options, modeling their behavior in traffic scenes is a challenging task. In this article we propose movement models based on machine learning methods, in particular artificial neural networks, in order to classify the current motion state and to predict the future trajectory of VRUs. Both…
In order to be globally deployed, autonomous cars must guarantee the safety of pedestrians. This is the reason why forecasting pedestrians' intentions sufficiently in advance is one of the most critical and challenging tasks for autonomous vehicles. This work tries to solve this problem by jointly predicting the intention and visual states of pedestrians. In terms of visual states, whereas previous work focused on x-y coordinates, we will also predict the size and indeed the whole bounding box of the pedestrian. The method is a recurrent neural network in a multi-task learning approach. It has…
EMT: A Visual Multi-Task Benchmark Dataset for Autonomous Driving
Computer Science, Khalifa University, Abu Dhabi, UAE（计算机科学，卡利法大学，阿布扎赫德，阿联酋）；Computer Science, New York University, Abu Dhabi, UAE（计算机科学，纽约大学，阿布扎赫德，阿联酋）；Computer and Information Engineering, Khalifa University, Abu Dhabi, UAE（计算机与信息工程，卡利法大学，阿布扎赫德，阿联酋）；KUCARS-KU Center for Autonomous Robotic Systems, Khalifa University, Abu Dhabi, UAE（KUCARS-KU自主机器人系统中心，卡利法大学，阿布扎赫德，阿联酋）
Carnegie Mellon University（卡内基梅隆大学）；University of California, Berkeley（加利福尼亚大学伯克利分校）
For autonomous vehicles (AVs) to behave appropriately on roads populated by human-driven vehicles, they must be able to reason about the uncertain intentions and decisions of other drivers from rich perceptual information. Towards these capabilities, we present a probabilistic forecasting model of future interactions between a variable number of agents. We perform both standard forecasting and the novel task of conditional forecasting, which reasons about how all agents will likely respond to the goal of a controlled agent (here, the AV). We train models on real and simulated data to forecast …
Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）
3D multi-object tracking (MOT) and trajectory forecasting are two critical components in modern 3D perception systems. We hypothesize that it is beneficial to unify both tasks under one framework to learn a shared feature representation of agent interaction. To evaluate this hypothesis, we propose a unified solution for 3D MOT and trajectory forecasting which also incorporates two additional novel computational units. First, we employ a feature interaction technique by introducing Graph Neural Networks (GNNs) to capture the way in which multiple agents interact with one another. The GNN is abl…
