arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2512.02458cs.CV
收藏

视觉到几何:用于连续具身MLLM推理和探索的3D空间记忆

Vision to Geometry: 3D Spatial Memory for Sequential Embodied MLLM Reasoning and Exploration

ACTION Lab, Michigan State University(密歇根州立大学ACTION实验室);SAIR Lab, University at Buffalo(布法罗大学SAIR实验室)

Zhongyi Cai, Yi Du, Chen Wang, Yu Kong

AI总结:

本文提出3DSPMR框架,利用视场覆盖作为几何先验,提升具身智能在连续任务中的记忆、推理与探索能力,并引入SEER-Bench基准测试连续具身探索与推理任务。

摘要:

具身智能体需通过主动探索未知环境并推理空间上下文来协助人类。在现实部署中,智能体常面临连续任务,其中每个新任务需在前一任务完成后再执行,可能包含不可行目标,如寻找不存在的物体。然而,现有研究多关注孤立目标,忽视了连续任务的核心挑战:重用先前探索积累的空间知识以指导后续推理与探索。本文研究这一未充分探索但实际重要的具身AI挑战。具体而言,我们提出3DSPMR,一种利用视场覆盖作为显式几何先验的3D空间记忆推理框架。通过整合基于视场的约束,3DSPMR显著增强了智能体在连续任务中的记忆、推理和探索能力。为促进该领域研究,我们进一步引入SEER-Bench,一个涵盖两个基础任务的新型连续具身探索与推理基准:具身问答(EQA)和具身多模态导航(EMN)。SEER-Bench独特地结合了可行与不可行任务,以提供对智能体性能的严格且全面的评估。大量实验验证了3DSPMR在连续EQA和EMN任务中实现了显著的性能提升。

英文摘要:

Embodied agents are expected to assist humans by actively exploring unknown environments and reasoning about spatial contexts. When deployed in real life, agents often face sequential tasks where each new task follows the completion of the previous one and may include infeasible objectives, such as searching for non-existent objects. However, most existing research focuses on isolated goals, overlooking the core challenge of sequential tasks: the ability to reuse spatial knowledge accumulated from previous explorations to guide subsequent reasoning and exploration. In this work, we investigate this underexplored yet practically significant embodied AI challenge. Specifically, we propose 3DSPMR, a 3D SPatial Memory Reasoning framework that utilizes Field-of-View (FoV) coverage as an explicit geometric prior. By integrating FoV-based constraints, 3DSPMR significantly enhances an agent's memory, reasoning, and exploration capabilities across sequential tasks. To facilitate research in this area, we further introduce SEER-Bench, a novel Sequential Embodied Exploration and Reasoning Benchmark that spans two foundational tasks: Embodied Question Answering (EQA) and Embodied Multi-modal Navigation (EMN). SEER-Bench uniquely incorporates both feasible and infeasible tasks to provide a rigorous and comprehensive evaluation of agent performance. Extensive experiments verify that 3DSPMR achieves substantial performance gains on both sequential EQA and EMN tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑