Embodied VideoAgent：来自第一人称视角视频和具身传感器的持久记忆实现动态场景理解
Embodied VideoAgent: Persistent Memory from Egocentric Videos and Embodied Sensors Enables Dynamic Scene Understanding
AI总结：
提出融合第一人称视频与深度、位姿等具身传感器的Embodied VideoAgent，通过VLM自动更新持久场景记忆，显著提升3D场景推理与规划性能。
摘要：
本文研究从第一人称视角观察中理解动态3D场景的问题，这是机器人学和具身AI中的一项关键挑战。不同于以往将其作为长视频理解来探索并仅利用第一人称视角视频的研究，我们转而提出一种基于LLM的智能体Embodied VideoAgent，它同时根据第一人称视角视频和具身感官输入（例如深度和位姿感知）构建场景记忆。我们进一步引入一种基于VLM的方法，用于在感知到针对物体的动作或活动时自动更新记忆。在3D场景中具有挑战性的推理和规划任务上，Embodied VideoAgent相比同类方法取得显著优势，在Ego4D-VQ3D、OpenEQA和EnvQA上分别实现4.9%、5.8%和11.7%的提升。我们还展示了其在多种具身AI任务中的潜力，包括生成具身交互以及用于机器人操作的感知。代码和演示将公开发布。
英文摘要：
This paper investigates the problem of understanding dynamic 3D scenes from egocentric observations, a key challenge in robotics and embodied AI. Unlike prior studies that explored this as long-form video understanding and utilized egocentric video only, we instead propose an LLM-based agent, Embodied VideoAgent, which constructs scene memory from both egocentric video and embodied sensory inputs (e.g. depth and pose sensing). We further introduce a VLM-based approach to automatically update the memory when actions or activities over objects are perceived. Embodied VideoAgent attains significant advantages over counterparts in challenging reasoning and planning tasks in 3D scenes, achieving gains of 4.9% on Ego4D-VQ3D, 5.8% on OpenEQA, and 11.7% on EnvQA. We have also demonstrated its potential in various embodied AI tasks including generating embodied interactions and perception for robot manipulation. The code and demo will be made public.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
