利用交叉注意力实现端到端的开放词汇全景重建
Leverage Cross-Attention for End-to-End Open-Vocabulary Panoptic Reconstruction
AI总结：
本文提出PanopticRecon++，通过新颖的交叉注意力机制实现端到端开放词汇全景重建，结合可学习3D高斯和最优线性分配，提升3D和2D分割重建性能。
摘要：
开放词汇全景重建提供了全面的场景理解，推动了具身机器人和逼真模拟的发展。在本文中，我们提出了PanopticRecon++，一种端到端的方法，通过新颖的交叉注意力视角来实现全景重建。该视角通过注意力图建模3D实例（作为查询）与场景3D嵌入场（作为键）之间的关系。与现有方法将查询和键的优化分开或忽略空间接近性不同，PanopticRecon++引入了可学习的3D高斯作为实例查询。这种建模方式注入了3D空间先验知识，以在保持端到端可优化性的同时保持接近性。此外，这种查询形式通过利用最优线性分配，利用从查询中渲染的实例掩码，促进了不同帧之间的2D开放词汇实例ID对齐。此外，我们通过一种由全景损失监督的新的全景头，将基于查询的实例分割概率与语义概率融合，以确保语义-实例分割的一致性。在训练过程中，实例查询标记的数量会动态适应以匹配物体数量。PanopticRecon++在模拟和现实世界数据集上在3D和2D分割和重建性能方面表现出竞争力，并展示了作为机器人模拟器的用户案例。我们的项目网站是：https://yuxuan1206.github.io/panopticrecon_pp/
英文摘要：
Open-vocabulary panoptic reconstruction offers comprehensive scene understanding, enabling advances in embodied robotics and photorealistic simulation. In this paper, we propose PanopticRecon++, an end-to-end method that formulates panoptic reconstruction through a novel cross-attention perspective. This perspective models the relationship between 3D instances (as queries) and the scene's 3D embedding field (as keys) through their attention map. Unlike existing methods that separate the optimization of queries and keys or overlook spatial proximity, PanopticRecon++ introduces learnable 3D Gaussians as instance queries. This formulation injects 3D spatial priors to preserve proximity while maintaining end-to-end optimizability. Moreover, this query formulation facilitates the alignment of 2D open-vocabulary instance IDs across frames by leveraging optimal linear assignment with instance masks rendered from the queries. Additionally, we ensure semantic-instance segmentation consistency by fusing query-based instance segmentation probabilities with semantic probabilities in a novel panoptic head supervised by a panoptic loss. During training, the number of instance query tokens dynamically adapts to match the number of objects. PanopticRecon++ shows competitive performance in terms of 3D and 2D segmentation and reconstruction performance on both simulation and real-world datasets, and demonstrates a user case as a robot simulator. Our project website is at: https://yuxuan1206.github.io/panopticrecon_pp/
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
