DINO Eats CLIP：超越已知领域进行开放集3D物体检索
DINO Eats CLIP: Adapting Beyond Knowns for Open-set 3D Object Retrieval
AI总结：
本文提出DEC框架，通过动态多视图整合和虚拟特征合成模块，提升开放集3D物体检索的性能，解决已知类别过拟合问题。
摘要：
视觉基础模型通过高效适应多视图图像在开放集3D物体检索（3DOR）中展现出巨大潜力。利用语义对齐的潜在空间，先前工作通常通过适应CLIP编码器构建基于视图的3D描述符。尽管CLIP具有强大的泛化能力，但其缺乏细粒度特性促使我们探索更近期的自监督编码器DINO的潜力。为此，我们提出DINO Eats CLIP（DEC），一种新的框架，通过合成未见类别的数据进行正则化，实现动态多视图整合。我们首先发现，对冻结DINO主干的视图特征进行简单均值池化可获得良好性能。然而，进一步适应会导致在已知类别的平均视图模式上严重过拟合。为解决此问题，我们设计了名为分块与适应模块（CAM）的模块，将多视图图像分成块并动态整合局部视图关系，产生比标准池化策略更稳健的特征。最后，我们提出虚拟特征合成（VFS）模块以显式缓解对已知类别的偏见。在底层，VFS利用CLIP的广泛、预对齐的视觉-语言空间合成虚拟特征用于未见类别。通过让DEC接触这些虚拟特征，我们大大增强了其开放集判别能力。在标准开放集3DOR基准上的广泛实验展示了其优越的效能。
英文摘要：
Vision foundation models have shown great promise for open-set 3D object retrieval (3DOR) through efficient adaptation to multi-view images. Leveraging semantically aligned latent space, previous work typically adapts the CLIP encoder to build view-based 3D descriptors. Despite CLIP's strong generalization ability, its lack of fine-grainedness prompted us to explore the potential of a more recent self-supervised encoder-DINO. To address this, we propose DINO Eats CLIP (DEC), a novel framework for dynamic multi-view integration that is regularized by synthesizing data for unseen classes. We first find that simply mean-pooling over view features from a frozen DINO backbone gives decent performance. Yet, further adaptation causes severe overfitting on average view patterns of known classes. To combat it, we then design a module named Chunking and Adapting Module (CAM). It segments multi-view images into chunks and dynamically integrates local view relations, yielding more robust features than the standard pooling strategy. Finally, we propose Virtual Feature Synthesis (VFS) module to mitigate bias towards known categories explicitly. Under the hood, VFS leverages CLIP's broad, pre-aligned vision-language space to synthesize virtual features for unseen classes. By exposing DEC to these virtual features, we greatly enhance its open-set discrimination capacity. Extensive experiments on standard open-set 3DOR benchmarks demonstrate its superior efficacy.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
