HexaGen3D:StableDiffusion 距离快速多样的文本到三维生成仅一步之遥
HexaGen3D: StableDiffusion is just one step away from Fast and Diverse Text-to-3D Generation
AI总结:
HexaGen3D 通过微调预训练二维扩散模型联合预测 6 个正交投影与潜在三平面,在 7 秒内实现无需逐样本优化的高质量、多样化文本到三维生成,并具备较强泛化能力。
摘要:
尽管生成建模近期取得了显著进展,但从文本提示高效生成高质量三维资产仍然是一项困难任务。一个关键挑战在于数据稀缺:最庞大的三维数据集仅包含数百万个资产,而其二维对应数据集则包含数十亿个文本-图像对。为解决这一问题,我们提出了一种利用大型预训练二维扩散模型能力的新方法。具体而言,我们的方法 HexaGen3D 对预训练的文本到图像模型进行微调,以联合预测 6 个正交投影及对应的潜在三平面。随后我们解码这些潜在表示以生成带纹理的网格。HexaGen3D 不需要逐样本优化,并能在 7 秒内从文本提示推断出高质量且多样的物体,与现有方法相比,提供了显著更好的质量-延迟权衡。此外,HexaGen3D 对新物体或组合展现出强大的泛化能力。
英文摘要:
Despite the latest remarkable advances in generative modeling, efficient generation of high-quality 3D assets from textual prompts remains a difficult task. A key challenge lies in data scarcity: the most extensive 3D datasets encompass merely millions of assets, while their 2D counterparts contain billions of text-image pairs. To address this, we propose a novel approach which harnesses the power of large, pretrained 2D diffusion models. More specifically, our approach, HexaGen3D, fine-tunes a pretrained text-to-image model to jointly predict 6 orthographic projections and the corresponding latent triplane. We then decode these latents to generate a textured mesh. HexaGen3D does not require per-sample optimization, and can infer high-quality and diverse objects from textual prompts in 7 seconds, offering significantly better quality-to-latency trade-offs when comparing to existing approaches. Furthermore, HexaGen3D demonstrates strong generalization to new objects or compositions.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。