3DStyle-Diffusion：利用2D扩散模型实现细粒度文本驱动的3D风格化
3DStyle-Diffusion: Pursuing Fine-grained Text-driven 3D Stylization with 2D Diffusion Models
AI总结：
针对文本驱动3D风格化的细粒度控制难题，提出3DStyle-Diffusion模型，结合2D扩散模型的几何与外观引导实现高质量细粒度3D网格风格化，还构建了对应数据集与评估协议。
摘要：
通过文本驱动的风格化进行3D内容创作，一直是多媒体和图形学领域面临的一项根本性挑战。跨模态基础模型（例如CLIP）的最新进展让这一问题的解决成为可能。这类方法通常利用CLIP将风格化网格的整体语义与给定的文本提示进行对齐。然而，仅依靠这种语义层面的跨模态监督，要实现3D网格细粒度细节的更可控风格化并非易事。\n在本研究中，我们提出了全新的3DStyle-Diffusion模型，该模型借助2D扩散模型提供的额外可控外观与几何引导，实现3D网格的细粒度风格化。在技术实现上，3DStyle-Diffusion首先通过隐式MLP（多层感知机）网络将3D网格的纹理参数化为反射属性与场景光照。同时，基于3D网格获取每个采样视角的精确深度图。随后，3DStyle-Diffusion利用预训练的可控2D扩散模型引导渲染图像的学习，促使每个视角的合成图像在语义上与文本提示对齐，在几何上与深度图保持一致。这种方式将基于隐式MLP网络的图像渲染与图像合成的扩散过程优雅地端到端整合，能够实现高质量的3D网格细粒度风格化。\n我们还基于Objaverse构建了一个新数据集，并制定了该任务的评估协议。通过定性与定量实验，我们验证了3DStyle-Diffusion的性能。源代码与数据集可访问https://github.com/yanghb22-fdu/3DStyle-Diffusion-Official获取。
英文摘要：
3D content creation via text-driven stylization has played a fundamental challenge to multimedia and graphics community. Recent advances of cross-modal foundation models (e.g., CLIP) have made this problem feasible. Those approaches commonly leverage CLIP to align the holistic semantics of stylized mesh with the given text prompt. Nevertheless, it is not trivial to enable more controllable stylization of fine-grained details in 3D meshes solely based on such semantic-level cross-modal supervision. In this work, we propose a new 3DStyle-Diffusion model that triggers fine-grained stylization of 3D meshes with additional controllable appearance and geometric guidance from 2D Diffusion models. Technically, 3DStyle-Diffusion first parameterizes the texture of 3D mesh into reflectance properties and scene lighting using implicit MLP networks. Meanwhile, an accurate depth map of each sampled view is achieved conditioned on 3D mesh. Then, 3DStyle-Diffusion leverages a pre-trained controllable 2D Diffusion model to guide the learning of rendered images, encouraging the synthesized image of each view semantically aligned with text prompt and geometrically consistent with depth map. This way elegantly integrates both image rendering via implicit MLP networks and diffusion process of image synthesis in an end-to-end fashion, enabling a high-quality fine-grained stylization of 3D meshes. We also build a new dataset derived from Objaverse and the evaluation protocol for this task. Through both qualitative and quantitative experiments, we validate the capability of our 3DStyle-Diffusion. Source code and data are available at \url{https://github.com/yanghb22-fdu/3DStyle-Diffusion-Official}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
