An Effective and Efficient Method for Detecting Hands in Egocentric Videos for Rehabilitation Applications
KITE – Toronto Rehab – University Health Network（KITE – 多伦多康复研究所 – 大学健康网络）；Rehabilitation Sciences Institute, University of Toronto（多伦多大学康复科学研究所）；Harvard Medical School, Harvard University（哈佛大学医学院）
Ryan J. Visée, Jirapat Likitlersuang, José Zariffa
Objective: Individuals with spinal cord injury (SCI) report upper limb function as their top recovery priority. To accurately represent the true impact of new interventions on patient function and independence, evaluation should occur in a natural setting. Wearable cameras can be used to monitor hand function at home, using computer vision to automatically analyze the resulting videos (egocentric video). A key step in this process, hand detection, is difficult to do robustly and reliably, hindering deployment of a complete monitoring system in the home and community. We propose an accurate and efficient hand detection method that uses a simple combination of existing detection and tracking algorithms. Methods: Detection, tracking, and combination methods were evaluated on a new hand detection dataset, consisting of 167,622 frames of egocentric videos collected on 17 individuals with SCI performing activities of daily living in a home simulation laboratory. Results: The F1-scores for the best detector and tracker alone (SSD and Median Flow) were 0.90$\pm$0.07 and 0.42$\pm$0.18, respectively. The best combination method, in which a detector was used to initialize and reset a tracker, resulted in an F1-score of 0.87$\pm$0.07 while being two times faster than the fastest detector alone. Conclusion: The combination of the fastest detector and best tracker improved the accuracy over online trackers while improving the speed of detectors. Significance: The method proposed here, in combination with wearable cameras, will help clinicians directly measure hand function in a patient's daily life at home, enabling independence after SCI.
University of Barcelona（巴塞罗那大学）；Computer Vision Centre（计算机视觉中心）
Recently, there has been a growing interest in analyzing human daily activities from data collected by wearable cameras. Since the hands are involved in a vast set of daily tasks, detecting hands in egocentric images is an important step towards the recognition of a variety of egocentric actions. However, besides extreme illumination changes in egocentric images, hand detection is not a trivial task because of the intrinsic large variability of hand appearance. We propose a hand detector that exploits skin modeling for fast hand proposal generation and Convolutional Neural Networks for hand re…
University of Barcelona（巴塞罗那大学）；Computer Vision Center（计算机视觉中心）
Given a user wearing a low frame rate wearable camera during a day, this work aims to automatically detect the moments when the user gets engaged into a social interaction solely by reviewing the automatically captured photos by the worn camera. The proposed method, inspired by the sociological concept of F-formation, exploits distance and orientation of the appearing individuals -with respect to the user- in the scene from a bird-view perspective. As a result, the interaction pattern over the sequence can be understood as a two-dimensional time series that corresponds to the temporal evolutio…
In a wearable camera video, we see what the camera wearer sees. While this makes it easy to know roughly what he chose to look at, it does not immediately reveal when he was engaged with the environment. Specifically, at what moments did his focus linger, as he paused to gather more information about something he saw? Knowing this answer would benefit various applications in video summarization and augmented reality, yet prior work focuses solely on the "what" question (estimating saliency, gaze) without considering the "when" (engagement). We propose a learning-based approach that uses long-t…
Detecting Activities of Daily Living in Egocentric Video to Contextualize Hand Use at Home in Outpatient Neurorehabilitation Settings
Institute of Biomedical Engineering, University of Toronto（多伦多大学生物医学工程研究所）；KITE - Toronto Rehabilitation Institute, University Health Network（大学健康网络KITE-多伦多康复研究所）；Rehabilitation Sciences Institute, University of Toronto（多伦多大学康复科学研究所）；Edward S. Rogers Sr. Department of Electrical and Computer Engineering, University of Toronto（多伦多大学爱德华·S·罗杰斯 Sr. 电气与计算机工程系）
University of Catania（卡塔尼亚大学）；Next Vision s.r.l.（Next Vision有限公司）
We consider the problem of detecting Egocentric HumanObject Interactions (EHOIs) in industrial contexts. Since collecting and labeling large amounts of real images is challenging, we propose a pipeline and a tool to generate photo-realistic synthetic First Person Vision (FPV) images automatically labeled for EHOI detection in a specific industrial scenario. To tackle the problem of EHOI detection, we propose a method that detects the hands, the objects in the scene, and determines which objects are currently involved in an interaction. We compare the performance of our method with a set of sta…
Samsung AI Center Cambridge（三星剑桥人工智能中心）；Adobe Research（奥多比研究院）
Egocentric videos capture sequences of human activities from a first-person perspective and can provide rich multimodal signals. However, most current localization methods use third-person videos and only incorporate visual information. In this work, we take a deep look into the effectiveness of audiovisual context in detecting actions in egocentric videos and introduce a simple-yet-effective approach via Observing, Watching, and Listening (OWL). OWL leverages audiovisual information and context for egocentric temporal action localization (TAL). We validate our approach in two large-scale data…
Augmented reality devices have the potential to enhance human perception and enable other assistive functionalities in complex conversational environments. Effectively capturing the audio-visual context necessary for understanding these social interactions first requires detecting and localizing the voice activities of the device wearer and the surrounding people. These tasks are challenging due to their egocentric nature: the wearer's head motion may cause motion blur, surrounding people may appear in difficult viewing angles, and there may be occlusions, visual clutter, audio noise, and bad …
SmartFactory-KL（凯泽斯劳滕智能工厂）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）
Detecting small objects in video streams of head-worn augmented reality devices in near real-time is a huge challenge: training data is typically scarce, the input video stream can be of limited quality, and small objects are notoriously hard to detect. In industrial scenarios, however, it is often possible to leverage contextual knowledge for the detection of small objects. Furthermore, CAD data of objects are typically available and can be used to generate synthetic training data. We describe a near real-time small object detection pipeline for egocentric perception in a manual assembly scen…
Measuring hand use in the home after cervical spinal cord injury using egocentric video
KITE Research Institute, University Health Network（KITE研究所（大学健康网络））；The BioRobotics Institute, Scuola Superiore Sant’Anna（圣安娜高等学校生物机器人研究所）；Institute of Biomedical Engineering, University of Toronto（多伦多大学生物医学工程研究所）；Sunnybrook Research Institute, Sunnybrook Health Sciences Centre（桑尼布鲁克健康科学中心桑尼布鲁克研究所）；Temerty Faculty of Medicine, University of Toronto（多伦多大学特默蒂医学院）；Brain and Spinal Cord Rehabilitation Program, Toronto Rehabilitation Institute – University Health Network（大学健康网络多伦多康复研究所脑与脊髓康复项目）；Department of Physical Therapy, University of Toronto（多伦多大学物理治疗系）；Edward S. Rogers Sr. Department of Electrical and Computer Engineering, University of Toronto（多伦多大学爱德华·S·罗杰斯 Sr. 电气与计算机工程系）
Understanding-Enhanced Model Collaboration for Long-Tailed Egocentric Mistake Detection
State Key Laboratory of AI Safety, Institute of Computing Technology, CAS（人工智能安全国家重点实验室，计算技术研究所，中国科学院）；School of Computer Science and Tech., University of Chinese Academy of Sciences（中国科学院大学计算机科学与技术学院）；Beijing Academy of Artificial Intelligence（北京人工智能研究院）；Institute of Information Engineering, CAS（信息工程研究所，中国科学院）；School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）
EgoAdapt: Enhancing Robustness in Egocentric Interactive Speaker Detection Under Missing Modalities
School of Computer and Communication Enngineering, University of Science and Technology Beijing（北京科技大学计算机与通信工程学院）；MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics（民航大学模式分析与机器智能重点实验室，计算机科学与技术学院）；MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, College of Computer Science and Technology, Shenzhen Research Institute, Nanjing University of Aeronautics and Astronautics（民航大学模式分析与机器智能重点实验室，计算机科学与技术学院，深圳研究院）
Detecting successful behaviour is crucial for training intelligent agents. As such, generalisable reward models are a prerequisite for agents that can learn to generalise their behaviour. In this work we focus on developing robust success detectors that leverage large, pretrained vision-language models (Flamingo, Alayrac et al. (2022)) and human reward annotations. Concretely, we treat success detection as a visual question answering (VQA) problem, denoted SuccessVQA. We study success detection across three vastly different domains: (i) interactive language-conditioned agents in a simulated ho…
