Object segmentation in infant's egocentric videos is a fundamental step in studying how children perceive objects in early stages of development. From the computer vision perspective, object segmentation in such videos pose quite a few challenges because the child's view is unfocused, often with large head movements, effecting in sudden changes in the child's point of view which leads to frequent change in object properties such as size, shape and illumination. In this paper, we develop a semi-automated, domain specific, method to address these concerns and facilitate the object annotation process for cognitive scientists allowing them to select and monitor the object under segmentation. The method starts with an annotation from the user of the desired object and employs graph cut segmentation and optical flow computation to predict the object mask for subsequent video frames automatically. To maintain accuracy, we use domain specific heuristic rules to re-initialize the program with new user input whenever object properties change dramatically. The evaluations demonstrate the high speed and accuracy of the presented method for object segmentation in voluminous egocentric videos. We apply the proposed method to investigate potential patterns in object distribution in child's view at progressive ages.
Indiana University（印第安纳大学）；The University of Texas at Austin（得克萨斯大学奥斯汀分校）
We analyze egocentric views of attended objects from infants. This paper shows 1) empirical evidence that children's egocentric views have more diverse distributions compared to adults' views, 2) we can computationally simulate the infants' distribution, and 3) the distribution is beneficial for training more generalized image classifiers not only for infant egocentric vision but for third-person computer vision.
Inspired by the remarkable ability of the infant visual learning system, a recent study collected first-person images from children to analyze the `training data' that they receive. We conduct a follow-up study that investigates two additional directions. First, given that infants can quickly learn to recognize a new object without much supervision (i.e. few-shot learning), we limit the number of training images. Second, we investigate how children control the supervision signals they receive during learning based on hand manipulation of objects. Our experimental results suggest that supervisi…
Luddy School of Informatics, Computing, and Engineering, Indiana University（印第安纳大学卢迪信息、计算与工程学院）；Department of Psychological and Brain Sciences, Indiana University（印第安纳大学心理与脑科学系）；Max Planck Institute for Intelligent Systems（马克斯·普朗克智能系统研究所）
Human infants have the remarkable ability to learn the associations between object names and visual objects from inherently ambiguous experiences. Researchers in cognitive science and developmental psychology have built formal models that implement in-principle learning algorithms, and then used pre-selected and pre-cleaned datasets to test the abilities of the models to find statistical regularities in the input data. In contrast to previous modeling approaches, the present study used egocentric video and gaze data collected from infant learners during natural toy play with their parents. Thi…
Reinforcing Egocentric Spatial Perception in Multimodal Large Language Models via Ego Scene Augmentation
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Guangxi Zhuang Autonomous Region Information Center（广西壮族自治区信息中心）；The Hong Kong University of Science and Technology（香港科技大学）
We present ongoing work to harness biological approaches to achieving highly efficient egocentric perception by combining the space-variant imaging architecture of the mammalian retina with Deep Learning methods. By pre-processing images collected by means of eye-tracking glasses to control the fixation locations of a software retina model, we demonstrate that we can reduce the input to a DCNN by a factor of 3, reduce the required number of training epochs and obtain over 98% classification rates when training and validating the system on a database of over 26,000 images of 9 object classes.
Nokia Bell Labs Spain（西班牙诺基亚贝尔实验室）；Universidad Autonoma de Madrid（马德里自治大学）
In this study, we focus on the egocentric segmentation of arms to improve self-perception in Augmented Virtuality (AV). The main contributions of this work are: i) a comprehensive survey of segmentation algorithms for AV; ii) an Egocentric Arm Segmentation Dataset, composed of more than 10, 000 images, comprising variations of skin color, and gender, among others. We provide all details required for the automated generation of groundtruth and semi-synthetic images; iii) the use of deep learning for the first time for segmenting arms in AV; iv) to showcase the usefulness of this database, we re…
PAWS: Perception of Articulation in the Wild at Scale from Egocentric Videos
Aalto University（阿尔托大学）；INSAIT, Sofia University（索菲亚大学INSAIT研究所）；ETH Zurich（苏黎世联邦理工学院）；TU Munich（慕尼黑工业大学）；MCML（慕尼黑机器学习中心）；ELLIS Institute Finland（芬兰ELLIS研究所）；University of Oulu（奥卢大学）
Egocentric perception has grown rapidly with the advent of immersive computing devices. Human gaze prediction is an important problem in analyzing egocentric videos and has primarily been tackled through either saliency-based modeling or highly supervised learning. We quantitatively analyze the generalization capabilities of supervised, deep learning models on the egocentric gaze prediction task on unseen, out-of-domain data. We find that their performance is highly dependent on the training data and is restricted to the domains specified in the training annotations. In this work, we tackle th…
EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Views
University of Science and Technology of China（中国科学技术大学）；Institute of Artificial Intelligence, Hefei Comprehensive National Science Center（合肥综合性国家科学中心人工智能研究院）
EgoAdapt: Adaptive Multisensory Distillation and Policy Learning for Efficient Egocentric Perception
University of Maryland, College Park（马里兰大学学院市分校）；Meta Reality Labs（Meta现实实验室）；Worcester Polytechnic Institute（沃斯特理工学院）；University of Toronto（多伦多大学）；FAIR, Meta AI（Meta AI）
提出 EgoAdapt 框架，通过自适应跨模态蒸馏与策略学习，在保持或超越 SOTA 性能的同时大幅降低多感官自我中心感知任务的计算与能耗成本。
SmartFactory-KL（凯泽斯劳滕智能工厂）；German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）
Detecting small objects in video streams of head-worn augmented reality devices in near real-time is a huge challenge: training data is typically scarce, the input video stream can be of limited quality, and small objects are notoriously hard to detect. In industrial scenarios, however, it is often possible to leverage contextual knowledge for the detection of small objects. Furthermore, CAD data of objects are typically available and can be used to generate synthetic training data. We describe a near real-time small object detection pipeline for egocentric perception in a manual assembly scen…
BinoGen: Scaling egocentric binocular data for embodied visual perception and learning
College of Biological Sciences, China Agricultural University（中国农业大学生物学院）；Beijing Institute for Brain Research, Chinese Academy of Medical Sciences & Peking Union Medical College（中国医学科学院北京协和医学院北京脑研究所）；Chinese Institute for Brain Research, Beijing (CIBR)（北京脑科学与类脑研究中心）
We present JRDB, a novel egocentric dataset collected from our social mobile manipulator JackRabbot. The dataset includes 64 minutes of annotated multimodal sensor data including stereo cylindrical 360$^\circ$ RGB video at 15 fps, 3D point clouds from two Velodyne 16 Lidars, line 3D point clouds from two Sick Lidars, audio signal, RGB-D video at 30 fps, 360$^\circ$ spherical image from a fisheye camera and encoder values from the robot's wheels. Our dataset incorporates data from traditionally underrepresented scenes such as indoor environments and pedestrian areas, all from the ego-perspectiv…
EgoInteract: Synthetic Egocentric Videos Generation for Interaction Understanding and Anticipation
Department of Mathematics and Computer Science, University of Catania（卡塔尼亚大学数学与计算机科学系）；Next Vision s.r.l.（Next Vision公司）；Reality Labs Research, Meta（Meta现实实验室）
Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understanding
State Key Laboratory of General Artificial Intelligence, BIGAI（通用人工智能国家重点实验室）；University of Science and Technology of China（中国科学技术大学）；Tsinghua University（清华大学）；Peking University（北京大学）
Assessing the alignment between infants' visual and linguistic experience using multimodal language models
Department of Psychology, Stanford University（心理学系，斯坦福大学）；Department of Psychology, University of California, San Diego（心理学系，加州大学圣地亚哥分校）；Department of Psychology, Tsinghua University（心理学系，清华大学）
Stanford University（斯坦福大学）；NVIDIA Research（英伟达研究院）；The University of Hong Kong（香港大学）
The ability to forecast human-environment collisions from egocentric observations is vital to enable collision avoidance in applications such as VR, AR, and wearable assistive robotics. In this work, we introduce the challenging problem of predicting collisions in diverse environments from multi-view egocentric videos captured from body-mounted cameras. Solving this problem requires a generalizable perception system that can classify which human body joints will collide and estimate a collision region heatmap to localize collisions in the environment. To achieve this, we propose a transformer-…
Humans have an innate ability to sense their surroundings, as they can extract the spatial representation from the egocentric perception and form an allocentric semantic map via spatial transformation and memory updating. However, endowing mobile agents with such a spatial sensing ability is still a challenge, due to two difficulties: (1) the previous convolutional models are limited by the local receptive field, thus, struggling to capture holistic long-range dependencies during observation; (2) the excessive computational budgets required for success, often lead to a separation of the mappin…
