VCD-Texture：面向文本引导纹理生成的基于方差对齐的3D-2D协同去噪
VCD-Texture: Variance Alignment based 3D-2D Co-Denoising for Text-Guided Texturing
AI总结：
针对现有3D纹理合成方法存在的模态鸿沟问题，提出基于方差对齐的3D-2D协同去噪框架VCD-Texture，同时构建新评估集，实验证明其性能优于同类方法。
摘要：
近期3D形状纹理合成研究得益于2D文本生成图像扩散模型的迅猛发展，相关方法包括基于修复的和基于优化的两类。但这些方法忽略了2D扩散模型与3D物体之间的模态鸿沟，它们主要将3D物体渲染为2D图像后分别对每张图像进行纹理生成。本文重新审视纹理合成任务，提出一种基于方差对齐的3D-2D协同去噪框架，命名为VCD-Texture，以解决上述问题。形式上，我们首先借助重投影的3D注意力感受野，在扩散自注意力模块中统一2D与3D隐特征学习。随后，去噪后的多视角2D隐特征被聚合到3D空间，再经光栅化回传以生成更具一致性的2D预测。然而光栅化过程存在难以处理的方差偏差，本文提出的方差对齐方法从理论上解决了该问题，实现了高保真纹理合成。此外，我们提出一种修复细化策略，进一步提升存在冲突区域的细节效果。值得注意的是，目前尚无公开基准用于评估纹理合成效果，这阻碍了该领域的发展。因此我们基于三个开源3D数据集构建了一个新的评估集，并提出采用四项指标全面验证纹理生成性能。综合实验表明，VCD-Texture相比其他同类方法取得了更优的性能。
英文摘要：
Recent research on texture synthesis for 3D shapes benefits a lot from dramatically developed 2D text-to-image diffusion models, including inpainting-based and optimization-based approaches. However, these methods ignore the modal gap between the 2D diffusion model and 3D objects, which primarily render 3D objects into 2D images and texture each image separately. In this paper, we revisit the texture synthesis and propose a Variance alignment based 3D-2D Collaborative Denoising framework, dubbed VCD-Texture, to address these issues. Formally, we first unify both 2D and 3D latent feature learning in diffusion self-attention modules with re-projected 3D attention receptive fields. Subsequently, the denoised multi-view 2D latent features are aggregated into 3D space and then rasterized back to formulate more consistent 2D predictions. However, the rasterization process suffers from an intractable variance bias, which is theoretically addressed by the proposed variance alignment, achieving high-fidelity texture synthesis. Moreover, we present an inpainting refinement to further improve the details with conflicting regions. Notably, there is not a publicly available benchmark to evaluate texture synthesis, which hinders its development. Thus we construct a new evaluation set built upon three open-source 3D datasets and propose to use four metrics to thoroughly validate the texturing performance. Comprehensive experiments demonstrate that VCD-Texture achieves superior performance against other counterparts.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
