Egocentric visual query localization is vital for embodied AI and VR/AR, yet remains challenging due to camera motion, viewpoint changes, and appearance variations. We present EAGLE, a novel framework that leverages episodic appearance- and geometry-aware memory to achieve unified 2D-3D visual query localization in egocentric vision. Inspired by avian memory consolidation, EAGLE synergistically integrates segmentation guided by an appearance-aware meta-learning memory (AMM), with tracking driven by a geometry-aware localization memory (GLM). This memory consolidation mechanism, through structured appearance and geometry memory banks, stores high-confidence retrieval samples, effectively supporting both long- and short-term modeling of target appearance variations. This enables precise contour delineation with robust spatial discrimination, leading to significantly improved retrieval accuracy. Furthermore, by integrating the VQL-2D output with a visual geometry grounded Transformer (VGGT), we achieve a efficient unification of 2D and 3D tasks, enabling rapid and accurate back-projection into 3D space. Our method achieves state-ofthe-art performance on the Ego4D-VQ benchmark.
CapMem: A Benchmark for Caption-Based Episodic Memory in Egocentric Video
University College London（伦敦大学学院）；Imperial College London（帝国理工学院）；Durham University（杜伦大学）；Shanghai Jiao Tong University（上海交通大学）；Queen Mary University of London（伦敦玛丽女王大学）
Georgia Tech（佐治亚理工学院）；Meta Reality Labs Research（Meta现实实验室）；Meta AI Research（Meta人工智能研究部门）
Egocentric augmented reality devices such as wearable glasses passively capture visual data as a human wearer tours a home environment. We envision a scenario wherein the human communicates with an AI agent powering such a device by asking questions (e.g., where did you last see my keys?). In order to succeed at this task, the egocentric AI assistant must (1) construct semantically rich and efficient scene memories that encode spatio-temporal information about objects seen during the tour and (2) possess the ability to understand the question and ground its answer into the semantic memory repr…
Enhancing Vision-Language Navigation with Multimodal Event Knowledge from Real-World Indoor Tour Videos
The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））；Tsinghua University（清华大学）；Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI)（马尔代夫 bin Zayed 大学人工智能学院）；Hangzhou City University（杭州城市学院）
OSGNet with MLLM Reranking @ Ego4D Episodic Memory Challenge 2026
Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））；National University of Singapore（新加坡国立大学）；Pengcheng Laboratory（鹏城实验室）；Shandong Jianzhu University（山东建筑大学）；Southern University of Science and Technology（南方科技大学）
UT Austin（德克萨斯大学奥斯汀分校）；University of Utah（犹他大学）；FAIR, Meta AI（Meta人工智能基础研究实验室）
Searching long egocentric videos with natural language queries (NLQ) has compelling applications in augmented reality and robotics, where a fluid index into everything that a person (agent) has seen before could augment human memory and surface relevant information on demand. However, the structured nature of the learning problem (free-form text query inputs, localized video temporal window outputs) and its needle-in-a-haystack nature makes it both technically challenging and expensive to supervise. We introduce Narrations-as-Queries (NaQ), a data augmentation strategy that transforms standard…
Meta AI；King Abdullah University of Science and Technology（阿卜杜拉国王科技大学）；University of Surrey（萨里大学）
The recently released Ego4D dataset and benchmark significantly scales and diversifies the first-person visual perception data. In Ego4D, the Visual Queries 2D Localization task aims to retrieve objects appeared in the past from the recording in the first-person view. This task requires a system to spatially and temporally localize the most recent appearance of a given object query, where query is registered by a single tight visual crop of the object in a different scene. Our study is based on the three-stage baseline introduced in the Episodic Memory benchmark. The baseline solves the proble…
SpaMEM: Benchmarking Dynamic Spatial Reasoning via Perception-Memory Integration in Embodied Environments
The University of New South Wales（新南威尔士大学）；The University of Alabama at Birmingham（阿拉巴马大学伯明翰分校）；Fudan University（复旦大学）；Tsinghua University（清华大学）；Zhejiang University（浙江大学）；Xinjiang University（新疆大学）；The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））
Vector Institute for AI（向量人工智能研究所）；Meta AI；University of British Columbia（不列颠哥伦比亚大学）
Video understanding tasks take many forms, from action detection to visual query localization and spatio-temporal grounding of sentences. These tasks differ in the type of inputs (only video, or video-query pair where query is an image region or sentence) and outputs (temporal segments or spatio-temporal tubes). However, at their core they require the same fundamental understanding of the video, i.e., the actors and objects in it, their actions and interactions. So far these tasks have been tackled in isolation with individual, highly specialized architectures, which do not exploit the interpl…
