用于快速三维语义提升的向量量化特征场
Vector Quantized Feature Fields for Fast 3D Semantic Lifting
AI总结：
本文提出向量量化特征场，通过轻量级按需检索像素对齐相关性掩码实现语义提升，在复杂室内外场景中验证有效性，并展示了其在文本驱动局部场景编辑和具身问答效率提升中的应用。
摘要：
我们通过引入逐视图掩码将提升推广为语义提升，这些掩码指示与提升任务相关的像素。这些掩码通过查询相应的多尺度像素对齐特征图来确定，这些特征图源自场景表示，如蒸馏特征场和特征点云。然而，存储从蒸馏特征场渲染的逐视图特征图是不切实际的，而特征点云的存储和查询成本高昂。为了实现像素对齐相关性掩码的轻量级按需检索，我们引入了向量量化特征场。我们在复杂的室内和室外场景上展示了向量量化特征场的有效性。语义提升与向量量化特征场相结合，可以解锁场景表示和具身智能中的众多应用。具体而言，我们展示了我们的方法如何实现文本驱动的局部场景编辑，并显著提高具身问答的效率。
英文摘要：
We generalize lifting to semantic lifting by incorporating per-view masks that indicate relevant pixels for lifting tasks. These masks are determined by querying corresponding multiscale pixel-aligned feature maps, which are derived from scene representations such as distilled feature fields and feature point clouds. However, storing per-view feature maps rendered from distilled feature fields is impractical, and feature point clouds are expensive to store and query. To enable lightweight on-demand retrieval of pixel-aligned relevance masks, we introduce the Vector-Quantized Feature Field. We demonstrate the effectiveness of the Vector-Quantized Feature Field on complex indoor and outdoor scenes. Semantic lifting, when paired with a Vector-Quantized Feature Field, can unlock a myriad of applications in scene representation and embodied intelligence. Specifically, we showcase how our method enables text-driven localized scene editing and significantly improves the efficiency of embodied question answering.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
