Falls are significant and often fatal for vulnerable populations such as the elderly. Previous works have addressed the detection of falls by relying on data capture by a single sensor, images or accelerometers. In this work, we rely on multimodal descriptors extracted from videos captured by egocentric cameras. Our proposed method includes a late decision fusion layer that builds on top of the extracted descriptors. Furthermore, we collect a new dataset on which we assess our proposed approach. We believe this is the first public dataset of its kind. The dataset comprises 10,948 video samples by 14 subjects. We conducted ablation experiments to assess the performance of individual feature extractors, fusion of visual information, and fusion of both visual and audio information. Moreover, we experimented with internal and external cross-validation. Our results demonstrate that the fusion of audio and visual information through late decision fusion improves detection performance, making it a promising tool for fall prevention and mitigation.
Facebook AI Research (FAIR)（脸书人工智能研究院）；University of Texas at Austin（德克萨斯大学奥斯汀分校）；International Institute of Information Technology, Hyderabad（海得拉巴国际信息技术学院）；Georgia Tech（佐治亚理工学院）；Carnegie Mellon University（卡内基梅隆大学）；University of Minnesota（明尼苏达大学）；Facebook（脸书）；University of Bristol（布里斯托尔大学）；Carnegie Mellon University Africa（卡内基梅隆大学非洲分校）；Universidad de los Andes（安第斯大学）；Facebook Reality Labs（脸书现实实验室）；University of Tokyo（东京大学）；Indiana University（印第安纳大学）
We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite. It offers 3,670 hours of daily-life activity video spanning hundreds of scenarios (household, outdoor, workplace, leisure, etc.) captured by 931 unique camera wearers from 74 worldwide locations and 9 different countries. The approach to collection is designed to uphold rigorous privacy and ethics standards with consenting participants and robust de-identification procedures where relevant. Ego4D dramatically expands the volume of diverse egocentric video footage publicly available to the research community. Port…
University of Barcelona（巴塞罗那大学）；Telefonica I+D（西班牙电信研发公司）；Institut de Robòtica i Informàtica Industrial (CSIC-UPC)（工业机器人与信息技术研究所（西班牙国家研究委员会-加泰罗尼亚理工大学））
Sounds are an important source of information on our daily interactions with objects. For instance, a significant amount of people can discern the temperature of water that it is being poured just by using the sense of hearing. However, only a few works have explored the use of audio for the classification of object interactions in conjunction with vision or as single modality. In this preliminary work, we propose an audio model for egocentric action recognition and explore its usefulness on the parts of the problem (noun, verb, and action classification). Our model achieves a competitive resu…
EgoIntrospect: An Egocentric Dataset and Benchmark for User-Centric Internal State Reasoning
Tsinghua University（清华大学）；Tongji University（同济大学）；Renmin University of China（中国人民大学）；The University of Tokyo（东京大学）；Lenovo Group（联想集团）；Peking University（北京大学）；University of Wisconsin–Madison（威斯康星大学麦迪逊分校）；Shanghai Qi Zhi Institute（上海启智研究院）
We present JRDB, a novel egocentric dataset collected from our social mobile manipulator JackRabbot. The dataset includes 64 minutes of annotated multimodal sensor data including stereo cylindrical 360$^\circ$ RGB video at 15 fps, 3D point clouds from two Velodyne 16 Lidars, line 3D point clouds from two Sick Lidars, audio signal, RGB-D video at 30 fps, 360$^\circ$ spherical image from a fisheye camera and encoder values from the robot's wheels. Our dataset incorporates data from traditionally underrepresented scenes such as indoor environments and pedestrian areas, all from the ego-perspectiv…
University of Bristol（布里斯托大学）；University of Oxford（牛津大学）
We focus on multi-modal fusion for egocentric action recognition, and propose a novel architecture for multi-modal temporal-binding, i.e. the combination of modalities within a range of temporal offsets. We train the architecture with three modalities -- RGB, Flow and Audio -- and combine them with mid-level fusion alongside sparse temporal sampling of fused representations. In contrast with previous works, modalities are fused before temporal aggregation, with shared modality and fusion weights over time. Our proposed architecture is trained end-to-end, outperforming individual modalities as …
Egocentric activity recognition in first-person videos has an increasing importance with a variety of applications such as lifelogging, summarization, assisted-living and activity tracking. Existing methods for this task are based on interpretation of various sensor information using pre-determined weights for each feature. In this work, we propose a new framework for egocentric activity recognition problem based on combining audio-visual features with multi-kernel learning (MKL) and multi-kernel boosting (MKBoost). For that purpose, firstly grid optical-flow, virtual-inertia feature, log-cova…
Samsung AI Center Cambridge（三星剑桥人工智能中心）；Adobe Research（奥多比研究院）
Egocentric videos capture sequences of human activities from a first-person perspective and can provide rich multimodal signals. However, most current localization methods use third-person videos and only incorporate visual information. In this work, we take a deep look into the effectiveness of audiovisual context in detecting actions in egocentric videos and introduce a simple-yet-effective approach via Observing, Watching, and Listening (OWL). OWL leverages audiovisual information and context for egocentric temporal action localization (TAL). We validate our approach in two large-scale data…
Facebook Reality Labs Research（Facebook现实实验室研究中心）；Facebook AI Applied Research（Facebook人工智能应用研究中心）
Augmented Reality (AR) as a platform has the potential to facilitate the reduction of the cocktail party effect. Future AR headsets could potentially leverage information from an array of sensors spanning many different modalities. Training and testing signal processing and machine learning algorithms on tasks such as beam-forming and speech enhancement require high quality representative data. To the best of the author's knowledge, as of publication there are no available datasets that contain synchronized egocentric multi-channel audio and video with dynamic movement and conversations in a n…
Audio-Sync Video Generation with Multi-Stream Temporal Control
Beijing Academy of Artificial Intelligence（北京人工智能研究院）；School of Software and Microelectronics, Peking University（北京大学软件与微电子学院）；School of Artificial Intelligence, Beijing University of Posts and Telecommunications（北京邮电大学人工智能学院）；Nat’l Key Lab of General AI, School of Intelligence Science and Technology, Peking University（国家通用人工智能实验室，北京大学智能科学与技术学院）；Nat’l Eng. Research Ctr. of Visual Tech., School of Computer Science, Peking University（国家视觉技术工程研究中心，北京大学计算机学院）
本文提出 MTV 框架，通过分离语音、音效和音乐多流音频实现细粒度音视频同步控制，并构建 DEMIX 数据集，在六项指标上达到最先进性能。
University of Rochester（罗切斯特大学）；Meta Reality Labs Research（Meta现实实验室）
Humans naturally perceive surrounding scenes by unifying sound and sight in a first-person view. Likewise, machines are advanced to approach human intelligence by learning with multisensory inputs from an egocentric perspective. In this paper, we explore the challenging egocentric audio-visual object localization task and observe that 1) egomotion commonly exists in first-person recordings, even within a short duration; 2) The out-of-view sound components can be created while wearers shift their attention. To address the first problem, we propose a geometry-aware temporal aggregation module to…
Epic-Sounds: A Large-scale Dataset of Actions That Sound
Visual Geometry Group, Department of Engineering Science, University of Oxford, UK（牛津大学视觉几何组）；Department of Computer Science, University of Bristol, UK（布里斯托大学计算机科学系）；CIIRC, Czech Technical University in Prague, Czech Republic（布拉格捷克技术大学CIIRC）
MacJEPA: Missingness-Robust Audio-Visual Recognition from Untrimmed Egocentric Videos
Peter L. Reichertz Institute for Medical Informatics, Hannover Medical School（汉诺威医学院彼得·L·赖歇茨医学信息学研究所）；Lower Saxony Center for Artificial Intelligence and Causal Methods in Medicine (CAIMed)（下萨克森州医学人工智能与因果方法中心（CAIMed））
Augmented reality devices have the potential to enhance human perception and enable other assistive functionalities in complex conversational environments. Effectively capturing the audio-visual context necessary for understanding these social interactions first requires detecting and localizing the voice activities of the device wearer and the surrounding people. These tasks are challenging due to their egocentric nature: the wearer's head motion may cause motion blur, surrounding people may appear in difficult viewing angles, and there may be occlusions, visual clutter, audio noise, and bad …
University of Barcelona（巴塞罗那大学）；Telefonica I+D, Research（西班牙电信研发研究部）；Institut de Robòtica i Informàtica Industrial (CSIC-UPC)（机器人与工业信息研究所（CSIC-UPC））
Our interaction with the world is an inherently multimodal experience. However, the understanding of human-to-object interactions has historically been addressed focusing on a single modality. In particular, a limited number of works have considered to integrate the visual and audio modalities for this purpose. In this work, we propose a multimodal approach for egocentric action recognition in a kitchen environment that relies on audio and visual information. Our model combines a sparse temporal sampling strategy with a late fusion of audio, spatial, and temporal streams. Experimental results …
Beijing University of Posts and Telecommunications（北京邮电大学）；Nanjing University（南京大学）
In this report, we present the transferring pretrained video mask autoencoders(VideoMAE) to egocentric tasks for Ego4d Looking at me Challenge. VideoMAE is the data-efficient pretraining model for self-supervised video pre-training and can easily transfer to downstream tasks. We show that the representation transferred from VideoMAE has good Spatio-temporal modeling and the ability to capture small actions. We only need to use egocentric data to train 10 epochs based on VideoMAE which pretrained by the ordinary videos acquired from a third person's view, and we can get better results than the …
Egocentric visual context detection can support intelligence augmentation applications. We created a wearable system, called PAL, for wearable, personalized, and privacy-preserving egocentric visual context detection. PAL has a wearable device with a camera, heart-rate sensor, on-device deep learning, and audio input/output. PAL also has a mobile/web application for personalized context labeling. We used on-device deep learning models for generic object and face detection, low-shot custom face and context recognition (e.g., activities like brushing teeth), and custom context clustering (e.g., …
University of Bristol（布里斯托大学）；Google Research（谷歌研究院）
In egocentric videos, actions occur in quick succession. We capitalise on the action's temporal context and propose a method that learns to attend to surrounding actions in order to improve recognition performance. To incorporate the temporal context, we propose a transformer-based multimodal model that ingests video and audio as input modalities, with an explicit language model providing action sequence context to enhance the predictions. We test our approach on EPIC-KITCHENS and EGTEA datasets reporting state-of-the-art performance. Our ablations showcase the advantage of utilising temporal …
FMReward: Aligning and Evaluating Audio-Driven 3D Facial Animation with Human Preferences
Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University（上海交通大学图像通信与网络工程研究所）；USC-SJTU Institute of Cultural and Creative Industry, Shanghai Jiao Tong University（上海交通大学 USC-文化创意产业学院）；Institut Universitaire de France (IUF)（法国大学研究院）；University of Nantes（南特大学）
DreamAudio: Customized Text-to-Audio Generation with Diffusion Models
School of Computer Science and Electronic Engineering, University of Surrey（Surrey大学计算机科学与电子工程学院）；Department of Informatics, King’s College London（伦敦国王学院信息学院）；Seed Group, ByteDance Inc.（字节跳动Seed团队）
Carnegie Mellon University（卡内基梅隆大学）；University of California, Berkeley（加州大学伯克利分校）
In order for AI to be safely deployed in real-world scenarios such as hospitals, schools, and the workplace, it must be able to robustly reason about the physical world. Fundamental to this reasoning is physical common sense: understanding the physical properties and affordances of available objects, how they can be manipulated, and how they interact with other objects. Physical commonsense reasoning is fundamentally a multi-sensory task, since physical properties are manifested through multiple modalities - two of them being vision and acoustics. Our paper takes a step towards real-world phys…
