arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.16019cs.RO
收藏

AIR-Embodied:一种基于主动3DGS的高效交互与重建框架,结合具身大语言模型

AIR-Embodied: An Efficient Active 3DGS-based Interaction and Reconstruction Framework with Embodied Large Language Model

School of Electrical and Electronic Engineering, Nanyang Technological University(南洋理工大学电气与电子工程学院);Beihang University(北京航空航天大学)

Zhenghao Qi, Shenghai Yuan, Fen Liu, Haozhi Cao, Tianchen Deng, Jianfei Yang, Lihua Xie

AI总结:

提出AIR-Embodied框架,结合具身AI智能体与多模态语言模型,通过理解状态、规划任务和闭环推理的三阶段流程,显著提升了主动3DGS重建的效率与质量。

摘要:

近期3D重建与神经渲染的进展提升了高质量数字资产的创建能力,但现有方法难以泛化至不同物体形状、纹理和遮挡情况。尽管Next Best View (NBV) 规划和基于学习的方法提供了方案,但它们常受限于预定义标准,且无法以人类常识处理遮挡。为解决这些问题,我们提出AIR-Embodied,一种将具身AI智能体与大规模预训练多模态语言模型集成以改进主动3DGS重建的新框架。AIR-Embodied采用三阶段流程:通过多模态提示理解当前重建状态,结合视角选择和交互动作进行任务规划,并采用闭环推理确保准确执行。智能体根据计划与实际结果间的差异动态优化其动作。在虚拟和真实环境中的实验评估表明,AIR-Embodied显著提升了重建效率和质量,为主动3D重建中的挑战提供了稳健的解决方案。

英文摘要:

Recent advancements in 3D reconstruction and neural rendering have enhanced the creation of high-quality digital assets, yet existing methods struggle to generalize across varying object shapes, textures, and occlusions. While Next Best View (NBV) planning and Learning-based approaches offer solutions, they are often limited by predefined criteria and fail to manage occlusions with human-like common sense. To address these problems, we present AIR-Embodied, a novel framework that integrates embodied AI agents with large-scale pretrained multi-modal language models to improve active 3DGS reconstruction. AIR-Embodied utilizes a three-stage process: understanding the current reconstruction state via multi-modal prompts, planning tasks with viewpoint selection and interactive actions, and employing closed-loop reasoning to ensure accurate execution. The agent dynamically refines its actions based on discrepancies between the planned and actual outcomes. Experimental evaluations across virtual and real-world environments demonstrate that AIR-Embodied significantly enhances reconstruction efficiency and quality, providing a robust solution to challenges in active 3D reconstruction.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑