We introduce a novel task of reconstructing a time series of second-person 3D human body meshes from monocular egocentric videos. The unique viewpoint and rapid embodied camera motion of egocentric videos raise additional technical barriers for human body capture. To address those challenges, we propose a simple yet effective optimization-based approach that leverages 2D observations of the entire video sequence and human-scene interaction constraint to estimate second-person human poses, shapes, and global motion that are grounded on the 3D environment captured from the egocentric view. We conduct detailed ablation studies to validate our design choice. Moreover, we compare our method with the previous state-of-the-art method on human motion capture from monocular video, and show that our method estimates more accurate human-body poses and shapes under the challenging egocentric setting. In addition, we demonstrate that our approach produces more realistic human-scene interaction.
University of Groningen（格罗宁根大学）；University of Barcelona（巴塞罗那大学）；Computer Vision Center（计算机视觉中心）
Nowadays, there is an upsurge of interest in using lifelogging devices. Such devices generate huge amounts of image data; consequently, the need for automatic methods for analyzing and summarizing these data is drastically increasing. We present a new method for familiar scene recognition in egocentric videos, based on background pattern detection through automatically configurable COSFIRE filters. We present some experiments over egocentric data acquired with the Narrative Clip.
EARL: Towards a Unified Analysis-Guided Reinforcement Learning Framework for Egocentric Interaction Reasoning and Pixel Grounding
Department of Electrical and Electronic Engineering, The Hong Kong Polytechnic University, Hong Kong SAR（香港理工大学电子与电气工程系）；Division of Emerging Interdisciplinary Areas (EMIA), The Hong Kong University of Science and Technology, Hong Kong SAR（香港理工大学新兴跨学科领域研究中心）
CVC, Universitat Autònoma de Barcelona（巴塞罗那自治大学计算机视觉中心）；MICC, University of Florence（佛罗伦萨大学MICC）
Word spotting in natural scene images has many applications in scene understanding and visual assistance. In this paper we propose a technique to create and exploit an intermediate representation of images based on text attributes which are character probability maps. Our representation extends the concept of the Pyramidal Histogram Of Characters (PHOC) by exploiting Fully Convolutional Networks to derive a pixel-wise mapping of the character distribution within candidate word regions. We call this representation the Soft-PHOC. Furthermore, we show how to use Soft-PHOC descriptors for word spo…
University of Groningen（格罗宁根大学）；University of Barcelona（巴塞罗那大学）；University of Rovira i Virgili（罗维拉-威尔吉利大学）
Recent studies have shown that the environment where people eat can affect their nutritional behaviour. In this work, we provide automatic tools for a personalised analysis of a person's health habits by the examination of daily recorded egocentric photo-streams. Specifically, we propose a new automatic approach for the classification of food-related environments, that is able to classify up to 15 such scenes. In this way, people can monitor the context around their food intake in order to get an objective insight into their daily eating routine. We propose a model that classifies food-related…
Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understanding
State Key Laboratory of General Artificial Intelligence, BIGAI（通用人工智能国家重点实验室）；University of Science and Technology of China（中国科学技术大学）；Tsinghua University（清华大学）；Peking University（北京大学）
Reinforcing Egocentric Spatial Perception in Multimodal Large Language Models via Ego Scene Augmentation
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Guangxi Zhuang Autonomous Region Information Center（广西壮族自治区信息中心）；The Hong Kong University of Science and Technology（香港科技大学）
Listening with the Eyes: Benchmarking Egocentric Co-Speech Grounding across Space and Time
Beijing Jiaotong University（北京交通大学）；Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences (CASIA)（基础模型研究中心、自动化研究所、中国科学院（CASIA））；Tencent Robotics X（腾讯机器人X）；Department of Computer Science and Engineering, The Hong Kong University of Science and Technology (HKUST)（计算机科学与工程系、香港科学与技术大学（HKUST））；Harbin Institute of Technology, Shenzhen（哈尔滨工业大学深圳学院）
JD Explore Academy（京东探索研究院）；University of Science and Technology of China（中国科学技术大学）；Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）；The University of Sydney（悉尼大学）
Affordance grounding, a task to ground (i.e., localize) action possibility region in objects, which faces the challenge of establishing an explicit link with object parts due to the diversity of interactive affordance. Human has the ability that transform the various exocentric interactions to invariant egocentric affordance so as to counter the impact of interactive diversity. To empower an agent with such ability, this paper proposes a task of affordance grounding from exocentric view, i.e., given exocentric human-object interaction and egocentric object images, learning the affordance knowl…
University of Minnesota（明尼苏达大学）；Carnegie Mellon University（卡内基梅隆大学）
In this paper, we study a problem of egocentric scene understanding, i.e., predicting depths and surface normals from an egocentric image. Egocentric scene understanding poses unprecedented challenges: (1) due to large head movements, the images are taken from non-canonical viewpoints (i.e., tilted images) where existing models of geometry prediction do not apply; (2) dynamic foreground objects including hands constitute a large proportion of visual scenes. These challenges limit the performance of the existing models learned from large indoor datasets, such as ScanNet and NYUv2, which compris…
Max Planck Institute for Informatics（马克斯·普朗克信息学研究所）；Intel Visual Computing Institute（英特尔视觉计算研究院）；University of Bath（巴斯大学）
Marker-based and marker-less optical skeletal motion-capture methods use an outside-in arrangement of cameras placed around a scene, with viewpoints converging on the center. They often create discomfort by possibly needed marker suits, and their recording volume is severely restricted and often constrained to indoor scenes with controlled backgrounds. We therefore propose a new method for real-time, marker-less and egocentric motion capture which estimates the full-body skeleton pose from a lightweight stereo pair of fisheye cameras that are attached to a helmet or virtual-reality headset. It…
Hand Trajectory Fusion for Egocentric Natural Language Query Grounding
Grupo de Tratamiento de Imágenes (GTI), Information Processing and Telecommunications Center , ETSI Telecomunicación, Universidad Politécnica de Madrid, Spain（图像处理小组（GTI）、信息处理与电信中心、电信工程学院、马德里理工大学、西班牙）
University of Science and Technology of China（中国科学技术大学）；The University of Sydney（悉尼大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
We present EgoNeRF, a practical solution to reconstruct large-scale real-world environments for VR assets. Given a few seconds of casually captured 360 video, EgoNeRF can efficiently build neural radiance fields which enable high-quality rendering from novel viewpoints. Motivated by the recent acceleration of NeRF using feature grids, we adopt spherical coordinate instead of conventional Cartesian coordinate. Cartesian feature grid is inefficient to represent large-scale unbounded scenes because it has a spatially uniform resolution, regardless of distance from viewers. The spherical parameter…
LongEgoRefer: A Benchmark for Long-Form Egocentric Video Referring Expression Comprehension
Woven by Toyota, Japan（丰田公司，日本）；The University of Tokyo, Japan（东京大学，日本）；National Institute of Informatics, Japan（日本信息机构国家研究所）；Institute of Science Tokyo, Japan（东京科学研究所，日本）；NII LLMC, Japan（日本信息机构LLMC）；Kyoto University, Japan（京都大学，日本）
