As humans move around, performing their daily tasks, they are able to recall where they have positioned objects in their environment, even if these objects are currently out of their sight. In this paper, we aim to mimic this spatial cognition ability. We thus formulate the task of Out of Sight, Not Out of Mind - 3D tracking active objects using observations captured through an egocentric camera. We introduce a simple but effective approach to address this challenging problem, called Lift, Match, and Keep (LMK). LMK lifts partial 2D observations to 3D world coordinates, matches them over time using visual appearance, 3D location and interactions to form object tracks, and keeps these object tracks even when they go out-of-view of the camera. We benchmark LMK on 100 long videos from EPIC-KITCHENS. Our results demonstrate that spatial cognition is critical for correctly locating objects over short and long time scales. E.g., for one long egocentric video, we estimate the 3D location of 50 active objects. After 120 seconds, 57% of the objects are correctly localised by LMK, compared to just 33% by a recent 3D method for egocentric videos and 17% by a general 2D tracking method.
Spontaneous Spatial Cognition Emerges during Egocentric Video Viewing through Non-invasive BCI
Hangzhou Dianzi University（杭州电子科技大学）；Zhejiang University（浙江大学）；RIKEN Center for Advanced Intelligence Project（日本理化学研究所先进智能项目中心）；RIKEN（日本理化学研究所）；Systems Research Institute（系统研究所）；Polish Academy of Sciences（波兰科学院）；Nicolaus Copernicus University（尼古拉哥尔斯基大学）；University of Rome Sapienza（罗马萨皮恩扎大学）
Bridging Perspectives: A Survey on Cross-view Collaborative Intelligence with Egocentric-Exocentric Vision
State Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家重点实验室，南京大学）；University of Tokyo（东京大学）；Zhejiang University（浙江大学）；Fudan University（复旦大学）
AlloEgo-VLM: Disambiguating Allocentric and Egocentric Reference Frames in Vision-Language Models
National Yang Ming Chiao Tung University（国立阳明交通大学）；Institute of Computer Science and Engineering（工程与计算机科学学院）；College of Artificial Intelligence（人工智能学院）
针对VLMs理解空间语义时 allocentric 与 egocentric 参考框架的歧义问题，构建数据集AlloEgo-View并开发框架AlloEgo-VLM，经NVIDIA Isaac Sim平台验证其在具身机器人开放式物体搜索任务中的有效性。
Cognitive Science Program, University of California, Berkeley（加州大学伯克利分校认知科学项目）；Department of Electrical and Computer Engineering, University of California San Diego（加州大学圣地亚哥分校电气与计算机工程系）；School of Computer Science, Georgia Institute of Technology & Emory University（佐治亚理工学院计算机科学学院及埃默里大学）；Department of Computer Science, Johns Hopkins University（约翰霍普金斯大学计算机科学系）；Department of Cognitive Science, University of California San Diego（加州大学圣地亚哥分校认知科学系）；Equal Advising Department of Computer Science & Wilmer Eye Institute, Johns Hopkins University（约翰霍普金斯大学计算机科学系及威尔默眼科研究所）；Language Technologies Institute, Carnegie Mellon University（卡内基梅隆大学语言技术研究所）；Robotics Institute, Carnegie Mellon University（卡内基梅隆大学机器人研究所）；Weinberg Institute for Cognitive Science, University of Michigan（密歇根大学韦恩伯格认知科学研究所）
Ego-InBetween: Generating Object State Transitions in Ego-Centric Videos
Advanced Micro Devices, Inc.（先进微器件公司）；Dalian University of Technology（大连理工大学）；The Hong Kong University of Science and Technology（香港科技大学）；The Chinese University of Hong Kong（香港中文大学）；Xi’an Jiaotong University（西安交通大学）
UCLA Center for Vision Cognition Learning and Autonomy（加州大学洛杉矶分校视觉认知学习与自主中心）
Humans possess a unique social cognition capability; nonverbal communication can convey rich social information among agents. In contrast, such crucial social characteristics are mostly missing in the existing scene understanding literature. In this paper, we incorporate different nonverbal communication cues (e.g., gaze, human poses, and gestures) to represent, model, learn, and infer agents' mental states from pure visual inputs. Crucially, such a mental representation takes the agent's belief into account so that it represents what the true world state is and infers the beliefs in each agen…
Mind Meets Space: Rethinking Agentic Spatial Intelligence from a Neuroscience-inspired Perspective
School of EEE, Nanyang Technological University（南洋理工大学电子工程系）；Department of Civil Engineering, Tsinghua University（清华大学土木工程系）；Dr. B. R. Ambedkar National Institute of Technology, Jalandhar（B.R.阿姆贝卡尔国家理工学院，贾兰德赫）；KTH Royal Institute of Technology（皇家理工学院）；College of AI, Tsinghua University（清华大学人工智能学院）；CCDS, Nanyang Technological University（南洋理工大学CCDS）
