Diff-SBSR: 学习多模态特征增强的扩散模型用于零样本素描基础3D形状检索
Diff-SBSR: Learning Multimodal Feature-Enhanced Diffusion Models for Zero-Shot Sketch-Based 3D Shape Retrieval
AI总结：
本文提出利用文本到图像扩散模型解决零样本素描基础3D形状检索问题，通过多模态特征增强策略提升语义上下文捕捉能力，实验表明方法在公开基准上优于现有技术。
摘要：
本文首次探索了文本到图像扩散模型在零样本素描基础3D形状检索（ZS-SBSR）中的应用。现有素描基础3D形状检索方法在零样本设置中面临类别监督缺失和素描输入极稀疏的挑战。我们的关键见解是大规模预训练扩散模型具有开放词汇能力和强形状偏倚，适合零样本视觉检索。我们利用冻结的Stable Diffusion主干，从中间U-Net层提取并聚合素描和渲染3D视图的判别表示。扩散模型难以处理素描，因其极抽象和稀疏，且与自然图像存在显著领域差距。为避免昂贵的重新训练，我们引入多模态特征增强策略，通过CLIP提取互补的视觉和文本线索条件冻结扩散主干，显式增强语义上下文捕捉能力并聚焦于素描轮廓。具体而言，我们注入由预训练CLIP视觉编码器提取的全局和局部视觉特征，并通过结合可学习软提示和BLIP生成的硬文本描述增强文本指导。此外，我们采用Circle-T损失动态加强正例对吸引力，一旦负样本充分分离，从而适应素描噪声并实现更有效的素描-3D对齐。在两个公开基准上的大量实验表明，我们的方法在ZS-SBSR任务中始终优于现有最先进方法。
英文摘要：
This paper presents the first exploration of text-to-image diffusion models for zero-shot sketch-based 3D shape retrieval (ZS-SBSR). Existing sketch-based 3D shape retrieval methods struggle in zero-shot settings due to the absence of category supervision and the extreme sparsity of sketch inputs. Our key insight is that large-scale pretrained diffusion models inherently exhibit open-vocabulary capability and strong shape bias, making them well suited for zero-shot visual retrieval. We leverage a frozen Stable Diffusion backbone to extract and aggregate discriminative representations from intermediate U-Net layers for both sketches and rendered 3D views. Diffusion models struggle with sketches due to their extreme abstraction and sparsity, compounded by a significant domain gap from natural images. To address this limitation without costly retraining, we introduce a multimodal feature-enhanced strategy that conditions the frozen diffusion backbone with complementary visual and textual cues from CLIP, explicitly enhancing the ability of semantic context capture and concentrating on sketch contours. Specifically, we inject global and local visual features derived from a pretrained CLIP visual encoder, and incorporate enriched textual guidance by combining learnable soft prompts with hard textual descriptions generated by BLIP. Furthermore, we employ the Circle-T loss to dynamically strengthen positive-pair attraction once negative samples are sufficiently separated, thereby adapting to sketch noise and enabling more effective sketch-3D alignment. Extensive experiments on two public benchmarks demonstrate that our method consistently outperforms state-of-the-art approaches in ZS-SBSR.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
