基于多模态条件的3D感知图像生成与编辑
3D-aware Image Generation and Editing with Multi-modal Conditions
AI总结：
针对3D一致图像生成中形状与外观解耦不足及控制模态单一的问题，提出融合噪声、文本和参考图像的端到端3D感知GAN框架，实现多样化生成、文本属性编辑与参考风格迁移。
摘要：
从单一2D语义标签生成3D一致的图像是计算机图形学与计算机视觉领域中一项重要且具有挑战性的研究课题。尽管一些相关工作已在该领域取得显著进展，但现有大多数方法在形状与外观的解耦性能方面表现较差，并且缺乏多模态控制能力。本文提出一种新颖的端到端3D感知图像生成与编辑模型，融合多种类型的条件输入，包括纯噪声、文本和参考图像。一方面，我们深入研究3D生成对抗网络（GANs）的潜空间，并提出一种新颖的解耦策略，在生成过程中将外观特征与形状特征分离。另一方面，我们提出一个统一框架，用于在多模态条件下灵活完成图像生成与编辑任务。该方法能够利用不同噪声生成多样化图像，通过文本描述编辑属性，并通过给定参考RGB图像进行风格迁移。大量实验表明，所提方法在图像生成与编辑的定性和定量评估中均优于其他替代方法。
英文摘要：
3D-consistent image generation from a single 2D semantic label is an important and challenging research topic in computer graphics and computer vision. Although some related works have made great progress in this field, most of the existing methods suffer from poor disentanglement performance of shape and appearance, and lack multi-modal control. In this paper, we propose a novel end-to-end 3D-aware image generation and editing model incorporating multiple types of conditional inputs, including pure noise, text and reference image. On the one hand, we dive into the latent space of 3D Generative Adversarial Networks (GANs) and propose a novel disentanglement strategy to separate appearance features from shape features during the generation process. On the other hand, we propose a unified framework for flexible image generation and editing tasks with multi-modal conditions. Our method can generate diverse images with distinct noises, edit the attribute through a text description and conduct style transfer by giving a reference RGB image. Extensive experiments demonstrate that the proposed method outperforms alternative approaches both qualitatively and quantitatively on image generation and editing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
