OpenDAS：面向2D和3D分割的开放词汇域自适应
OpenDAS: Open-Vocabulary Domain Adaptation for 2D and 3D Segmentation
AI总结：
针对开放词汇分割在基础类别性能弱、现有自适应方法难保留开集能力的问题，提出OpenDAS开放词汇域自适应方法，结合参数高效提示调优与三元组损失策略，在2D、3D分割任务上均实现性能提升。
摘要：
近年来，视觉-语言模型（Vision-Language Models, VLMs）推动分割技术从传统的预定义类别闭集分割转向开放词汇分割（open-vocabulary segmentation, OVS），使用户能够分割分割模型训练期间未见过的新类别和新概念。然而，这种灵活性存在权衡：全监督闭集方法在基础类别（即它们被明确训练过的类别）上的性能仍然优于OVS方法。这是因为VLMs（基于图像-文本对训练）缺乏像素对齐的训练掩码，同时缺少自动驾驶等领域的特定知识。因此，我们提出开放词汇域自适应任务，在保留VLMs开放词汇特性的同时，将领域特定知识注入其中，从而提升基础类别和新类别的性能。现有的VLM自适应方法虽然能提升基础（训练）查询的性能，但无法完全保留VLMs在新查询上的开集能力。为解决这一缺陷，我们将参数高效的提示调优与基于三元组损失的训练策略相结合，该策略使用辅助负查询。值得注意的是，我们的方法是唯一一种在新类别上持续超越原始VLM的参数高效方法。我们自适应后的VLMs可以无缝集成到现有OVS流程中：例如，在开放词汇2D分割任务上，使OVSeg在ADE20K数据集上的mIoU提升+6.0%；在开放词汇3D实例分割任务上，使OpenMask3D在ScanNet++ Offices数据集上的AP提升+4.1%，且无需其他修改。项目页面位于https://open-das.github.io/。
英文摘要：
Recently, Vision-Language Models (VLMs) have advanced segmentation techniques by shifting from the traditional segmentation of a closed-set of predefined object classes to open-vocabulary segmentation (OVS), allowing users to segment novel classes and concepts unseen during training of the segmentation model. However, this flexibility comes with a trade-off: fully-supervised closed-set methods still outperform OVS methods on base classes, that is on classes on which they have been explicitly trained. This is due to the lack of pixel-aligned training masks for VLMs (which are trained on image-caption pairs), and the absence of domain-specific knowledge, such as autonomous driving. Therefore, we propose the task of open-vocabulary domain adaptation to infuse domain-specific knowledge into VLMs while preserving their open-vocabulary nature. By doing so, we achieve improved performance in base and novel classes. Existing VLM adaptation methods improve performance on base (training) queries, but fail to fully preserve the open-set capabilities of VLMs on novel queries. To address this shortcoming, we combine parameter-efficient prompt tuning with a triplet-loss-based training strategy that uses auxiliary negative queries. Notably, our approach is the only parameter-efficient method that consistently surpasses the original VLM on novel classes. Our adapted VLMs can seamlessly be integrated into existing OVS pipelines, e.g., improving OVSeg by +6.0% mIoU on ADE20K for open-vocabulary 2D segmentation, and OpenMask3D by +4.1% AP on ScanNet++ Offices for open-vocabulary 3D instance segmentation without other changes. The project page is available at https://open-das.github.io/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
