Egocentric Human-Object Interaction (EHOI) analysis is crucial for industrial safety, yet the development of robust models is hindered by the scarcity of annotated domain-specific data. We address this challenge by introducing a data generation framework that combines synthetic data with a diffusion-based process to augment real-world images with realistic Personal Protective Equipment (PPE). We present GlovEgo-HOI, a new benchmark dataset for industrial EHOI, and GlovEgo-Net, a model integrating Glove-Head and Keypoint- Head modules to leverage hand pose information for enhanced interaction detection. Extensive experiments demonstrate the effectiveness of the proposed data generation framework and GlovEgo-Net. To foster further research, we release the GlovEgo-HOI dataset, augmentation pipeline, and pre-trained models at: GitHub project.
HOIMotion: Forecasting Human Motion During Human-Object Interactions Using Egocentric 3D Object Bounding Boxes
University of Stuttgart（斯图加特大学）；Heidelberg University（海德堡大学）；University of Tuebingen（蒂宾根大学）；Center for Bionic Intelligence Tuebingen Stuttgart (BITS)（蒂宾根-斯图加特仿生智能中心（BITS））
EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Views
University of Science and Technology of China（中国科学技术大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
EgoGrasp: World-Space Hand-Object Interaction Estimation from Egocentric Videos
Shanghai Jiao Tong University（上海交通大学）；The University of Hong Kong（香港大学）；University of Science and Technology of China（中国科学技术大学）；Imperial College London（伦敦帝国学院）；Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；Beijing Academy of Artificial Intelligence（北京人工智能研究院）
In this paper, we present a method to detect the hand-object interaction from an egocentric perspective. In contrast to massive data-driven discriminator based method like \cite{Shan20}, we propose a novel workflow that utilises the cues of hand and object. Specifically, we train networks predicting hand pose, hand mask and in-hand object mask to jointly predict the hand-object interaction status. We compare our method with the most recent work from Shan et al. \cite{Shan20} on selected images from EPIC-KITCHENS \cite{damen2018scaling} dataset and achieve $89\%$ accuracy on HOI (hand-object in…
Egocentric World Model for Photorealistic Hand-Object Interaction Synthesis
Texas A&M University（德克萨斯A&M大学）；University of Minnesota（明尼苏达大学）；University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；University of California, Los Angeles（加州大学洛杉矶分校）；University of Texas at Austin（得克萨斯大学奥斯汀分校）；Amazon（亚马逊）；State University of New York at Stony Brook（纽约州立大学石溪分校）
EgoPhys: Estimating Peak Contact Force and Mechanical Work from Egocentric Manipulation Video
School of Artificial Intelligence, Shandong University（山东大学人工智能学院）；Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）；School of Mechanical Engineering, Tianjin University（天津大学机械工程学院）
FlowHOI: Flow-based Semantics-Grounded Generation of Hand-Object Interactions for Dexterous Robot Manipulation
Mohamed bin Zayed University of Artificial Intelligence（莫扎德·扎耶德人工智能大学）；Technical University of Munich（慕尼黑技术大学）；National University of Singapore（新加坡国立大学）；Westlake University（西湖大学）
University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）；Intel Labs（英特尔实验室）；UC San Diego（加州大学圣地亚哥分校）
We propose to forecast future hand-object interactions given an egocentric video. Instead of predicting action labels or pixels, we directly predict the hand motion trajectory and the future contact points on the next active object (i.e., interaction hotspots). This relatively low-dimensional representation provides a concrete description of future interactions. To tackle this task, we first provide an automatic way to collect trajectory and hotspots labels on large-scale data. We then use this data to train an Object-Centric Transformer (OCT) model for prediction. Our model performs hand and …
University of Bristol（布里斯托大学）；Phillips Research（飞利浦研究院）；Shanghai University（上海大学）
In this paper, we propose a method to jointly determine the status of hand-object interaction. This is crucial for egocentric human activity understanding and interaction. From a computer vision perspective, we believe that determining whether a hand is interacting with an object depends on whether there is an interactive hand pose and whether the hand is touching the object. Thus, we extract the hand pose, hand-object masks to jointly determine the interaction status. In order to solve the problem of hand pose estimation due to in-hand object occlusion, we use a multi-cam system to capture ha…
In this paper, we concern with the problem of how to automatically extract the steps that compose real-life hand activities. This is a key competence towards processing, monitoring and providing video guidance in Mixed Reality systems. We use egocentric vision to observe hand-object interactions in real-world tasks and automatically decompose a video into its constituent steps. Our approach combines hand-object interaction (HOI) detection, object similarity measurement and a finite state machine (FSM) representation to automatically edit videos into steps. We use a combination of Convolutional…
