在任何地方、任何时间描述任何事物
Describe Anything Anywhere At Any Moment
AI总结:
DAAAM是一种用于大规模实时4D场景理解的空间时间记忆框架,通过优化前端和分层4D场景图实现高效语义描述与实时性能的平衡。
摘要:
计算机视觉和机器人应用,从增强现实到大规模环境中的机器人自主性,都需要能够捕捉几何结构以实现准确语言定位以及语义细节的空间时间记忆框架。现有方法面临权衡,即产生丰富的开放式词汇描述会以牺牲实时性能为代价,当这些描述需要在3D空间中定位时。为了解决这些挑战,我们提出了Describe Anything, Anywhere, at Any Moment (DAAAM),一种新的空间时间记忆框架,用于大规模和实时的4D场景理解。DAAAM引入了一种基于优化的前端,从本地化描述模型(如Describe Anything Model (DAM))中推断出详细的语义描述,利用批量处理技术将推理速度提升一个数量级以适应在线处理。它利用这种语义理解构建一个分层的4D场景图(SG),该图作为有效的全局空间和时间一致的记忆表示。DAAAM在构建具有详细几何基础描述的4D SG的同时保持实时性能。我们展示了DAAAM的4D SG能够与用于推理和推理的工具调用代理良好对接。我们彻底评估了DAAAM在复杂的时空问答任务上的性能,即在NaVQA基准测试中展示了其在SG3D基准测试上的通用能力。我们进一步整理了一个扩展的OC-NaVQA基准测试,用于大规模和长时间评估。DAAAM在两项任务中均取得了最先进的结果,分别在OC-NaVQA问题准确性上提高了53.6%,位置误差减少了21.9%,时间误差减少了21.6%,以及SG3D任务定位准确性提高了27.8%。我们发布了我们的数据和代码开源。
英文摘要:
Computer vision and robotics applications ranging from augmented reality to robot autonomy in large-scale environments require spatio-temporal memory frameworks that capture both geometric structure for accurate language-grounding as well as semantic detail. Existing methods face a tradeoff, where producing rich open-vocabulary descriptions comes at the expense of real-time performance when these descriptions have to be grounded in 3D. To address these challenges, we propose Describe Anything, Anywhere, at Any Moment (DAAAM), a novel spatio-temporal memory framework for large-scale and real-time 4D scene understanding. DAAAM introduces a novel optimization-based frontend to infer detailed semantic descriptions from localized captioning models, such as the Describe Anything Model (DAM), leveraging batch processing to speed up inference by an order of magnitude for online processing. It leverages such semantic understanding to build a hierarchical 4D scene graph (SG), which acts as an effective globally spatially and temporally consistent memory representation. DAAAM constructs 4D SGs with detailed, geometrically grounded descriptions while maintaining real-time performance. We show that DAAAM's 4D SG interfaces well with a tool-calling agent for inference and reasoning. We thoroughly evaluate DAAAM in the complex task of spatio-temporal question answering on the NaVQA benchmark and show its generalization capabilities for sequential task grounding on the SG3D benchmark. We further curate an extended OC-NaVQA benchmark for large-scale and long-time evaluations. DAAAM achieves state-of-the-art results in both tasks, improving OC-NaVQA question accuracy by 53.6%, position errors by 21.9%, temporal errors by 21.6%, and SG3D task grounding accuracy by 27.8% over the most competitive baselines, respectively. We release our data and code open-source.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。