发表机构
University of Washington; Amazon; The University of Texas at Austin（华盛顿大学; 亚马逊公司; 德克萨斯大学奥斯汀分校）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本研究针对具身场景三维问答的效率问题，提出MemTree3D记忆树引导的关键帧选择方法，在OpenEQA数据集上显著提升GPT-4o与LLaVA-OneVision-7B的问答性能，优于现有视觉搜索方法。
AI 中文摘要
在具身场景中，由于视觉语言模型（VLM）推理的计算与内存资源有限，准确且高效地回答问题面临重大挑战。现有方法采用视觉搜索关键帧检索方法，为VLM输入选择与问题相关的关键帧，但该方法效率低下，因为需针对每个用户查询在数千个视频帧中进行视觉搜索。本研究提出一种记忆树引导的关键帧选择范式，用于具身场景中的高效三维问答。该方法利用名为MemTree3D的紧凑可复用三维场景表示，其支持利用相机6自由度位姿进行实时在线构建，可捕获多级三维场景信息，使大语言模型无需重新处理整个视频流，即可通过基于评分的帧选择高效查询并检索与问题相关的关键帧。在OpenEQA数据集上，本方法使GPT-4o的LLM-Match提升17.4%，使LLaVA-OneVision-7B的LLM-Match提升5.8%，优于现有视觉搜索方法，代码可在指定网址获取。
英文摘要
Answering questions accurately and efficiently in embodied scenarios presents significant challenges due to limited computational and memory resources for Vision Language Model (VLM) inference. Existing methods adopt visual search key frame retrieval method to select critical question-related key frames for VLM input. However, visual search methods are inefficient because they require visual search among thousands of video frames for each individual user query. In this work, we propose a memory tree guided key frame selection paradigm for efficient 3D question answering in embodied scenarios. Our method leverages a compact and reusable 3D scene representation, termed MemTree3D, which supports real-time online construction leveraging camera 6-DoF poses. MemTree3D captures multi-level 3D scene information, enabling a Large Language Model to efficiently query and retrieve question-relevant key frames through our scoring-based frame selection without reprocessing the entire video stream. On OpenEQA, our method improves the LLM-Match of GPT-4o by 17.4%, LLaVA-OneVision-7B by 5.8%, outperforms existing visual search methods. Our code is available at https://github.com/hsiangwei0903/MemTree3D
CommentsECCV 2026
