面向空间感知AI的多模态3D融合与原位学习
Multimodal 3D Fusion and In-Situ Learning for Spatially Aware AI
AI总结：
本研究提出融合CLIP特征的多模态3D表示与原位学习方法，搭建快速重建流水线，在Magic Leap 2上实现自然语言空间搜索、智能库存两个AR应用，开源代码以推动空间感知AI发展。
摘要：
增强现实中虚拟与物理世界的无缝融合，得益于系统对物理环境的语义级“理解”。AR研究长期聚焦于情境感知的潜力，展示了利用3D环境中的语义实现各类物体级交互的全新能力。与此同时，计算机视觉领域在神经视觉-语言理解方面取得了长足进展，提升了自主任务的环境感知能力。在本研究中，我们提出一种多模态3D物体表示方法，将语义知识、语言知识与几何表示相统一，支持用户引导的、涉及物理物体的机器学习。我们首先提出一种快速多模态3D重建流水线，通过将CLIP视觉-语言特征融合到环境与物体模型中，为AR赋予语言理解能力。随后我们提出“原位”机器学习方法，该方法与多模态表示相结合，为用户提供了全新的工具与交互界面，使其能以具备空间与语言意义的方式与物理空间及物体交互。我们通过在Magic Leap 2上的两个真实AR应用验证了所提系统的实用性：a) 物理环境中的自然语言空间搜索；b) 追踪物体随时间变化的智能库存系统。我们还在(https://github.com/cy-xu/spatially_aware_AI)公开了完整实现与演示数据，以推动空间感知AI领域的进一步探索与研究。
英文摘要：
Seamless integration of virtual and physical worlds in augmented reality benefits from the system semantically "understanding" the physical environment. AR research has long focused on the potential of context awareness, demonstrating novel capabilities that leverage the semantics in the 3D environment for various object-level interactions. Meanwhile, the computer vision community has made leaps in neural vision-language understanding to enhance environment perception for autonomous tasks. In this work, we introduce a multimodal 3D object representation that unifies both semantic and linguistic knowledge with the geometric representation, enabling user-guided machine learning involving physical objects. We first present a fast multimodal 3D reconstruction pipeline that brings linguistic understanding to AR by fusing CLIP vision-language features into the environment and object models. We then propose "in-situ" machine learning, which, in conjunction with the multimodal representation, enables new tools and interfaces for users to interact with physical spaces and objects in a spatially and linguistically meaningful manner. We demonstrate the usefulness of the proposed system through two real-world AR applications on Magic Leap 2: a) spatial search in physical environments with natural language and b) an intelligent inventory system that tracks object changes over time. We also make our full implementation and demo data available at (https://github.com/cy-xu/spatially_aware_AI) to encourage further exploration and research in spatially aware AI.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
