Bidirectional Progressive Transformer for Interaction Intention Anticipation
University of Science and Technology of China（中国科学技术大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
Zichen Zhang, Hongchen Luo, Wei Zhai, Yang Cao, Yu Kang
Interaction intention anticipation aims to jointly predict future hand trajectories and interaction hotspots. Existing research often treated trajectory forecasting and interaction hotspots prediction as separate tasks or solely considered the impact of trajectories on interaction hotspots, which led to the accumulation of prediction errors over time. However, a deeper inherent connection exists between hand trajectories and interaction hotspots, which allows for continuous mutual correction between them. Building upon this relationship, a novel Bidirectional prOgressive Transformer (BOT), which introduces a Bidirectional Progressive mechanism into the anticipation of interaction intention is established. Initially, BOT maximizes the utilization of spatial information from the last observation frame through the Spatial-Temporal Reconstruction Module, mitigating conflicts arising from changes of view in first-person videos. Subsequently, based on two independent prediction branches, a Bidirectional Progressive Enhancement Module is introduced to mutually improve the prediction of hand trajectories and interaction hotspots over time to minimize error accumulation. Finally, acknowledging the intrinsic randomness in human natural behavior, we employ a Trajectory Stochastic Unit and a C-VAE to introduce appropriate uncertainty to trajectories and interaction hotspots, respectively. Our method achieves state-of-the-art results on three benchmark datasets Epic-Kitchens-100, EGO4D, and EGTEA Gaze+, demonstrating superior in complex scenarios.
Numerous car accidents are caused by improper driving maneuvers. Serious injuries are however avoidable if such driving maneuvers are detected beforehand and the driver is assisted accordingly. In fact, various recent research has focused on the automated prediction of driving maneuver based on hand-crafted features extracted mainly from in-cabin driver videos. Since the outside view from the traffic scene may also contain informative features for driving maneuver prediction, we present a framework for the detection of the drivers' intention based on both in-cabin and traffic scene videos. Mor…
Intention Action Anticipation Model with Guide-Feedback Loop Mechanism
School of Physics and Electronic, Yan’an University（延安大学物理与电子学院）；College of Computer Science, Chongqing University（重庆大学计算机学院）；Nanyang Technological University（南洋理工大学）
Human intention prediction is a growing area of research where an activity in a video has to be anticipated by a vision-based system. To this end, the model creates a representation of the past, and subsequently, it produces future hypotheses about upcoming scenarios. In this work, we focus on pedestrians' early intention prediction in which, from a current observation of an urban scene, the model predicts the future activity of pedestrians that approach the street. Our method is based on a multi-modal transformer that encodes past observations and produces multiple predictions at different an…
Action-Dynamics Modeling and Cross-Temporal Interaction for Online Action Understanding
School of Computing and Mathematical Sciences, University of Leicester, United Kingdom（莱斯特大学计算与数学科学学院）；School of Eye and Vision Sciences, University of Liverpool, United Kingdom（利物浦大学眼科学与视觉科学学院）；School of Information Science and Engineering, University of Jinan, China（济南大学信息科学与工程学院）；School of Automation and Information Engineering, Xi’an University of Technology, China（西安理工大学自动化与信息工程学院）
RGB-D Video Generation for Improving Human-to-Robot Object Handover Prediction
Nanyang Technological University（南洋理工大学）；College of Computing and Data Science（计算与数据科学学院）；Zhejiang University（浙江大学）；College of Computer Science and Technology（计算机科学与技术学院）；Alibaba Group（阿里巴巴集团）
Inria（法国国家信息与自动化研究所）；CNRS（法国国家科学研究中心）；University of Grenoble - Alpes（格勒诺布尔-阿尔卑斯大学）
Recognition of intentions is a subconscious cognitive process vital to human communication. This skill enables anticipation and increases the quality of interactions between humans. Within the context of engagement, non-verbal signals are used to communicate the intention of starting the interaction with a partner. In this paper, we investigated methods to detect these signals in order to allow a robot to know when it is about to be addressed. Originality of our approach resides in taking inspiration from social and cognitive sciences to perform our perception task. We investigate meaningful f…
Georgia Institute of Technology（佐治亚理工学院）；ETH Zürich（苏黎世联邦理工学院）；University of Wisconsin-Madison（威斯康星大学麦迪逊分校）
We address the challenging task of anticipating human-object interaction in first person videos. Most existing methods ignore how the camera wearer interacts with the objects, or simply consider body motion as a separate modality. In contrast, we observe that the international hand movement reveals critical information about the future activity. Motivated by this, we adopt intentional hand movement as a future representation and propose a novel deep network that jointly models and predicts the egocentric hand motion, interaction hotspots and future action. Specifically, we consider the future …
Imperial College London（帝国理工学院）；Data Science Institute（数据科学研究所）
Eye movements have long been studied as a window into the attentional mechanisms of the human brain and made accessible as novelty style human-machine interfaces. However, not everything that we gaze upon, is something we want to interact with; this is known as the Midas Touch problem for gaze interfaces. To overcome the Midas Touch problem, present interfaces tend not to rely on natural gaze cues, but rather use dwell time or gaze gestures. Here we present an entirely data-driven approach to decode human intention for object manipulation tasks based solely on natural gaze cues. We run data co…
