不在视野内，就不在上下文中？VLM在跨不相交帧中的自我中心空间推理
Out of Sight, Not Out of Context? Egocentric Spatial Reasoning in VLMs Across Disjoint Frames
AI总结：
本文提出Disjoint-3DQA基准评估VLMs跨不相交帧的自我中心空间推理能力，发现模型在3D场景表示上存在瓶颈，提供oracle 3D坐标可显著提升性能。
摘要：
在自我中心视频上运行的具身AI助手必须跨时间整合空间线索——例如，确定片刻前瞥见的物体A相对于稍后遇到的物体B的位置。我们引入了Disjoint-3DQA，这是一个生成式QA基准，通过提出关于在同一帧中不可同见的物体对的问题来评估VLMs的这种能力。我们评估了七个最先进的VLMs，发现模型落后人类表现28%，且随着时间间隔扩大，准确率急剧下降（从60%降至30%）。我们的分析进一步揭示，向VLMs提供轨迹或鸟瞰图投影仅带来微小的改进，而提供oracle 3D坐标则带来20%的显著性能提升。这突显了多帧VLMs在从视觉信号随时间构建和维护3D场景表示方面的核心瓶颈。因此，Disjoint-3DQA为长期空间推理设定了清晰、可衡量的挑战，旨在推动视觉、语言和具身AI交叉领域的未来研究。
英文摘要：
An embodied AI assistant operating on egocentric video must integrate spatial cues across time - for instance, determining where an object A, glimpsed a few moments ago lies relative to an object B encountered later. We introduce Disjoint-3DQA , a generative QA benchmark that evaluates this ability of VLMs by posing questions about object pairs that are not co-visible in the same frame. We evaluated seven state-of-the-art VLMs and found that models lag behind human performance by 28%, with steeper declines in accuracy (60% to 30 %) as the temporal gap widens. Our analysis further reveals that providing trajectories or bird's-eye-view projections to VLMs results in only marginal improvements, whereas providing oracle 3D coordinates leads to a substantial 20% performance increase. This highlights a core bottleneck of multi-frame VLMs in constructing and maintaining 3D scene representations over time from visual signals. Disjoint-3DQA therefore sets a clear, measurable challenge for long-horizon spatial reasoning and aims to catalyze future research at the intersection of vision, language, and embodied AI.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
