SpatialMem: 用于语言定位和问答的度量对齐长周期视频记忆
SpatialMem: Metric-Aligned Long-Horizon Video Memory for Language Grounding and QA
AI总结：
SpatialMem通过构建度量对齐的空间支架，实现长周期视频记忆的可解释检索与问答，支持语言引导的检索和离线导航任务。
摘要：
我们提出了SpatialMem，一种以记忆为中心的系统，用于从第一人称视频中进行长周期、语言基础的检索和问答，其中度量3D充当可解释的索引支架，而不是显式的映射目标。从随意拍摄的第一人称RGB视频开始，SpatialMem构建了一个度量对齐的空间支架，用于室内场景，检测结构3D锚点（墙壁、门、窗户）作为第一层支持，并构建一个层次化记忆，其中开放词汇对象节点将证据片段、视觉嵌入和两层文本描述链接到3D坐标，以实现紧凑存储和快速检索。这种设计使可解释、空间基础的查询关系（例如距离、方向、可见性）成为可能，并支持下游任务，如语言引导的检索/问答和在预构建的记忆上进行离线导航式指导，而无需专用传感器。在Replica场景和两个真实世界的第一人称室内场景上的实验表明，尽管增加的杂乱和遮挡，SpatialMem在这些评估场景中保持稳定的布局推理、离线指导和层次化检索。一个紧凑的消融进一步表明，两层描述记忆提高了路径级定位，而适度的尺度扰动只会导致有限的降级。这些结果将SpatialMem定位为一种高效且可扩展的空间基础长周期视频理解的记忆接口。
英文摘要：
We present SpatialMem, a memory-centric system for long-horizon, language-grounded retrieval and QA from egocentric video, where metric 3D serves as an interpretable indexing scaffold rather than an explicit mapping objective. Starting from casually captured egocentric RGB video, SpatialMem builds a metric-aligned spatial scaffold for indoor scenes, detects structural 3D anchors (walls, doors, windows) as first-layer support, and populates a hierarchical memory with open-vocabulary object nodes that link evidence patches, visual embeddings, and two-layer textual descriptions to 3D coordinates for compact storage and fast retrieval. This design enables interpretable, spatially grounded queries over relations (e.g., distance, direction, visibility) and supports downstream tasks such as language-guided retrieval/QA and offline navigation-style guidance over a prebuilt memory, without specialized sensors. Experiments on one public Replica scene and two real-world egocentric indoor scenes show that SpatialMem maintains stable layout reasoning, offline guidance, and hierarchical retrieval across these evaluated scenes despite increasing clutter and occlusion. A compact ablation further shows that the two-layer description memory improves path-level grounding, while moderate scale perturbation causes only limited degradation. These results position SpatialMem as an efficient and extensible memory interface for spatially grounded long-horizon video understanding.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
