AIR-Embodied:一种基于主动3DGS的高效交互与重建框架,结合具身大语言模型
AIR-Embodied: An Efficient Active 3DGS-based Interaction and Reconstruction Framework with Embodied Large Language Model
AI总结:
提出AIR-Embodied框架,结合具身AI智能体与多模态语言模型,通过理解状态、规划任务和闭环推理的三阶段流程,显著提升了主动3DGS重建的效率与质量。
摘要:
近期3D重建与神经渲染的进展提升了高质量数字资产的创建能力,但现有方法难以泛化至不同物体形状、纹理和遮挡情况。尽管Next Best View (NBV) 规划和基于学习的方法提供了方案,但它们常受限于预定义标准,且无法以人类常识处理遮挡。为解决这些问题,我们提出AIR-Embodied,一种将具身AI智能体与大规模预训练多模态语言模型集成以改进主动3DGS重建的新框架。AIR-Embodied采用三阶段流程:通过多模态提示理解当前重建状态,结合视角选择和交互动作进行任务规划,并采用闭环推理确保准确执行。智能体根据计划与实际结果间的差异动态优化其动作。在虚拟和真实环境中的实验评估表明,AIR-Embodied显著提升了重建效率和质量,为主动3D重建中的挑战提供了稳健的解决方案。
英文摘要:
Recent advancements in 3D reconstruction and neural rendering have enhanced the creation of high-quality digital assets, yet existing methods struggle to generalize across varying object shapes, textures, and occlusions. While Next Best View (NBV) planning and Learning-based approaches offer solutions, they are often limited by predefined criteria and fail to manage occlusions with human-like common sense. To address these problems, we present AIR-Embodied, a novel framework that integrates embodied AI agents with large-scale pretrained multi-modal language models to improve active 3DGS reconstruction. AIR-Embodied utilizes a three-stage process: understanding the current reconstruction state via multi-modal prompts, planning tasks with viewpoint selection and interactive actions, and employing closed-loop reasoning to ensure accurate execution. The agent dynamically refines its actions based on discrepancies between the planned and actual outcomes. Experimental evaluations across virtual and real-world environments demonstrate that AIR-Embodied significantly enhances reconstruction efficiency and quality, providing a robust solution to challenges in active 3D reconstruction.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。