形状注入的联合嵌入能否改进图像条件下的3D扩散？
Can Shape-Infused Joint Embeddings Improve Image-Conditioned 3D Diffusion?
AI总结：
本研究提出CISP，通过将2D图像与3D形状对齐的联合嵌入来增强图像条件下的3D扩散生成，在保持生成质量与多样性的同时，显著提升生成形状与输入图像的一致性。
摘要：
深度生成模型的最新进展，特别是将CLIP（对比语言-图像预训练）应用于去噪扩散概率模型（DDPMs），已在文本到图像生成中展现出显著效果。CLIP结构良好的嵌入空间也被扩展到使用DDPMs进行图像到形状的生成，并取得了显著成果。尽管取得了这些成功，一些基本问题随之浮现：CLIP能否确保从图像生成形状的最佳结果？我们能否利用条件化将显式的3D知识引入生成过程并获得更高质量？本研究提出了CISP（对比图像-形状预训练），旨在增强由2D图像引导的3D形状合成。CISP旨在通过将2D图像与3D形状在共享嵌入空间中对齐来丰富CLIP框架，特别捕捉CLIP以文本-图像为中心可能忽略的3D特征。我们的综合分析评估了CISP相对于CLIP引导模型的引导性能，重点关注生成质量、多样性以及生成形状与条件图像的一致性。我们发现，虽然CISP在生成质量和多样性上与CLIP相当，但它显著提高了与输入图像的一致性，强调了将3D知识融入生成模型的价值。这些发现为通过将多模态系统与3D表示相结合来推进3D视觉内容合成指明了一个有前景的方向。
英文摘要：
Recent advancements in deep generative models, particularly with the application of CLIP (Contrastive Language Image Pretraining) to Denoising Diffusion Probabilistic Models (DDPMs), have demonstrated remarkable effectiveness in text to image generation. The well structured embedding space of CLIP has also been extended to image to shape generation with DDPMs, yielding notable results. Despite these successes, some fundamental questions arise: Does CLIP ensure the best results in shape generation from images? Can we leverage conditioning to bring explicit 3D knowledge into the generative process and obtain better quality? This study introduces CISP (Contrastive Image Shape Pre training), designed to enhance 3D shape synthesis guided by 2D images. CISP aims to enrich the CLIP framework by aligning 2D images with 3D shapes in a shared embedding space, specifically capturing 3D characteristics potentially overlooked by CLIP's text image focus. Our comprehensive analysis assesses CISP's guidance performance against CLIP guided models, focusing on generation quality, diversity, and coherence of the produced shapes with the conditioning image. We find that, while matching CLIP in generation quality and diversity, CISP substantially improves coherence with input images, underscoring the value of incorporating 3D knowledge into generative models. These findings suggest a promising direction for advancing the synthesis of 3D visual content by integrating multimodal systems with 3D representations.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
