CORE-3D: 基于嵌入的上下文感知开放词汇检索
CORE-3D: Context-aware Open-vocabulary Retrieval by Embeddings in 3D
AI总结：
CORE-3D通过结合上下文感知的CLIP编码和渐进粒度细化，提升3D场景理解中开放词汇检索和语义分割的准确性与鲁棒性。
摘要：
3D场景理解是具身AI和机器人领域的重要基础，支持可靠感知以实现交互和导航。最近的方法通过将嵌入向量分配给通过视觉-语言模型（VLMs）生成的2D类无关掩码，并将其投影到3D中，实现了零样本、开放词汇的3D语义映射。然而，这些方法由于直接使用原始掩码，往往会产生碎片化的掩码和不准确的语义分配，限制了其在复杂环境中的有效性。为了解决这一问题，我们利用SemanticSAM结合渐进性粒度细化来生成更准确且数量更多的对象级掩码，缓解在掩码生成模型如vanilla SAM中常见的过度分割问题，并提高下游的3D语义分割。为进一步增强语义上下文，我们采用一种上下文感知的CLIP编码策略，通过经验确定的加权整合每个掩码的多个上下文视图，提供更丰富的视觉上下文。我们在多个3D场景理解任务上评估了我们的方法，包括3D语义分割和从语言查询中检索对象，跨多个基准数据集。实验结果表明，与现有方法相比有显著改进，突显了我们方法的有效性。
英文摘要：
Object retrieval from a scene has become a new trend of research due to its numerous applications. Recent approaches achieve zero-shot, open-vocabulary 3D semantic mapping by assigning embedding vectors to 2D class-agnostic masks generated via vision-language models (VLMs) and projecting these into 3D. However, these methods often produce fragmented masks and inaccurate semantic assignments due to the direct use of raw masks, limiting their effectiveness in complex environments. To address this, we leverage SemanticSAM with progressive granularity refinement to generate more accurate and numerous object-level masks, mitigating the over-segmentation commonly observed in mask generation models such as vanilla SAM, and improving downstream 3D semantic segmentation. To further enhance semantic context, we employ a context-aware CLIP encoding strategy that integrates multiple contextual views of each mask using empirically determined weighting, providing much richer visual context. We evaluate our approach on multiple 3D scene understanding tasks, including 3D semantic segmentation and object retrieval from language queries, across several benchmark datasets. Experimental results demonstrate significant improvements over existing methods, highlighting the effectiveness of our approach.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
