First-person video naturally brings the use of a physical environment to the forefront, since it shows the camera wearer interacting fluidly in a space based on his intentions. However, current methods largely separate the observed actions from the persistent space itself. We introduce a model for environment affordances that is learned directly from egocentric video. The main idea is to gain a human-centric model of a physical space (such as a kitchen) that captures (1) the primary spatial zones of interaction and (2) the likely activities they support. Our approach decomposes a space into a topological map derived from first-person activity, organizing an ego-video into a series of visits to the different zones. Further, we show how to link zones across multiple related environments (e.g., from videos of multiple kitchens) to obtain a consolidated representation of environment functionality. On EPIC-Kitchens and EGTEA+, we demonstrate our approach for learning scene affordances and anticipating future actions in long-form video.
Industrial Institute of Science, The University of Tokyo（东京大学产业科学研究所）
Object affordance is an important concept in human-object interaction, providing information on action possibilities based on human motor capacity and objects' physical property thus benefiting tasks such as action anticipation and robot imitation learning. However, existing datasets often: 1) mix up affordance with object functionality; 2) confuse affordance with goal-related action; and 3) ignore human motor capacity. This paper proposes an efficient annotation scheme to address these issues by combining goal-irrelevant motor actions and grasp types as affordance labels and introducing the c…
Industrial Institute of Science, The University of Tokyo（东京大学生产技术研究所）
Object affordance is an important concept in hand-object interaction, providing information on action possibilities based on human motor capacity and objects' physical property thus benefiting tasks such as action anticipation and robot imitation learning. However, the definition of affordance in existing datasets often: 1) mix up affordance with object functionality; 2) confuse affordance with goal-related action; and 3) ignore human motor capacity. This paper proposes an efficient annotation scheme to address these issues by combining goal-irrelevant motor actions and grasp types as affordan…
University of Science and Technology of China（中国科学技术大学）；The University of Sydney（悉尼大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
A long-standing goal of intelligent assistants such as AR glasses/robots has been to assist users in affordance-centric real-world scenarios, such as "how can I run the microwave for 1 minute?". However, there is still no clear task definition and suitable benchmarks. In this paper, we define a new task called Affordance-centric Question-driven Task Completion, where the AI assistant should learn from instructional videos to provide step-by-step help in the user's view. To support the task, we constructed AssistQ, a new dataset comprising 531 question-answer samples from 100 newly filmed instr…
JD Explore Academy（京东探索研究院）；University of Science and Technology of China（中国科学技术大学）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）；The University of Sydney（悉尼大学）
Affordance grounding, a task to ground (i.e., localize) action possibility region in objects, which faces the challenge of establishing an explicit link with object parts due to the diversity of interactive affordance. Human has the ability that transform the various exocentric interactions to invariant egocentric affordance so as to counter the impact of interactive diversity. To empower an agent with such ability, this paper proposes a task of affordance grounding from exocentric view, i.e., given exocentric human-object interaction and egocentric object images, learning the affordance knowl…
In this abstract we describe recent [4,7] and latest work on the determination of affordances in visually perceived 3D scenes. Our method builds on the hypothesis that geometry on its own provides enough information to enable the detection of significant interaction possibilities in the environment. The motivation behind this is that geometric information is intimately related to the physical interactions afforded by objects in the world. The approach uses a generic representation for the interaction between everyday objects such as a mug or an umbrella with the environment, and also for more …
Integrating Affordances and Attention models for Short-Term Object Interaction Anticipation
Aragon Institute for Engineering Research (I3A)（阿拉贡工程研究所）；University of Zaragoza（萨拉戈塔大学）；Department of Computer Science（计算机科学系）；University of Catania（卡塔尼亚大学）
UT Austin（德克萨斯大学奥斯汀分校）；Facebook AI Research（Facebook人工智能研究院）
Embodied agents operating in human spaces must be able to master how their environment works: what objects can the agent use, and how can it use them? We introduce a reinforcement learning approach for exploration for interaction, whereby an embodied agent autonomously discovers the affordance landscape of a new unmapped 3D environment (such as an unfamiliar kitchen). Given an egocentric RGB-D camera and a high-level action space, the agent is rewarded for maximizing successful interactions while simultaneously training an image-based affordance segmentation model. The former yields a policy f…
MADiff: Motion-Aware Mamba Diffusion Models for Hand Trajectory Prediction on Egocentric Videos
IRMV Lab, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University and State Key Laboratory of Avionics Integration and Aviation System-of-Systems Synthesis, Shanghai Key Laboratory of Navigation and Location Based Services（IRMV实验室，自动化与智能感知学院，上海交通大学，航空集成与航空系统-of-Systems综合国家重点实验室，导航与定位服务重点实验室）；College of Intelligence Science and Technology, National University of Defense Technology（智能科学与技术学院，国防科技大学）；Department of Electronic Engineering, Shanghai Jiao Tong University（电子工程学院，上海交通大学）
One-Shot Affordance Grounding of Deformable Objects in Egocentric Organizing Scenes
School of Artificial Intelligence and Robotics, Hunan University, China（人工智能与机器人学院，湖南大学）；National Engineering Research Center of Robot Visual Perception and Control Technology, Hunan University, China（机器人视觉感知与控制技术国家工程研究中心，湖南大学）
Diff-IP2D: Diffusion-Based Hand-Object Interaction Prediction on Egocentric Videos
IRMV Lab, the Department of Automation, Shanghai Jiao Tong University, Shanghai 200240, China（IRMV实验室，自动化系，上海交通大学，上海200240，中国）；College of Intelligence Science and Technology, National University of Defense Technology, Changsha 410073, China（智能科学与技术学院，国防科技大学，长沙410073，中国）；Department of Electronic Engineering, Shanghai Jiao Tong University, Shanghai 200240, China（电子工程系，上海交通大学，上海200240，中国）
This work tackles scene understanding for outdoor robotic navigation, solely relying on images captured by an on-board camera. Conventional visual scene understanding interprets the environment based on specific descriptive categories. However, such a representation is not directly interpretable for decision-making and constrains robot operation to a specific domain. Thus, we propose to segment egocentric images directly in terms of how a robot can navigate in them, and tailor the learning problem to an autonomous navigation task. Building around an image segmentation network, we present a gen…
AffordanceWAM: Affordance-Aware Joint World-Action Modeling for Robot Manipulation
The University of Hong Kong（香港大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Peking University（北京大学）；The Chinese University of Hong Kong（香港中文大学）；National University of Singapore（新加坡国立大学）；Knowin AI（考拉悠然科技）
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）
Interactive object understanding, or what we can do to objects and how is a long-standing goal of computer vision. In this paper, we tackle this problem through observation of human hands in in-the-wild egocentric videos. We demonstrate that observation of what human hands interact with and how can provide both the relevant data and the necessary supervision. Attending to hands, readily localizes and stabilizes active objects for learning and reveals places where interactions with objects occur. Analyzing the hands shows what we can do to objects and how. We apply these basic principles on the…
ETH Zurich（苏黎世联邦理工学院）；Microsoft（微软）；Samsung AI Center, Cambridge（三星AI中心（剑桥））
We present a comprehensive framework for egocentric interaction recognition using markerless 3D annotations of two hands manipulating objects. To this end, we propose a method to create a unified dataset for egocentric 3D interaction recognition. Our method produces annotations of the 3D pose of two hands and the 6D pose of the manipulated objects, along with their interaction labels for each frame. Our dataset, called H2O (2 Hands and Objects), provides synchronized multi-view RGB-D images, interaction labels, object classes, ground-truth 3D poses for left & right hands, 6D object poses, grou…
We explore the problem of intersection classification using monocular on-board passive vision, with the goal of classifying traffic scenes with respect to road topology. We divide the existing approaches into two broad categories according to the type of input data: (a) first person vision (FPV) approaches, which use an egocentric view sequence as the intersection is passed; and (b) third person vision (TPV) approaches, which use a single view immediately before entering the intersection. The FPV and TPV approaches each have advantages and disadvantages. Therefore, we aim to combine them into …
