arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2505.06166cs.CV
收藏

DiffLocks:利用扩散模型从单张图像生成三维头发

DiffLocks: Generating 3D Hair from a Single Image using Diffusion Models

Meshcapade;Zhejiang University(浙江大学);Stanford University(斯坦福大学);Max Planck Institute for Intelligent Systems(马克斯·普朗克智能系统研究所)

Radu Alexandru Rosu, Keyu Wu, Yao Feng, Youyi Zheng, Michael J. Black

AI总结:

DiffLocks通过构建含4万发型的最大合成数据集并训练图像条件扩散变换器,首次实现从单张图像直接重建包括非洲式卷发在内的多样化精细三维发丝。

摘要:

我们致力于解决从单张图像生成三维头发几何结构的任务,该任务因发型多样性以及缺乏成对的图像到三维头发数据而极具挑战性。以往的方法主要在合成数据上训练,并通过使用低维中间表示(如引导发丝和头皮级嵌入)来应对此类数据量有限的问题,但这些表示需要后处理才能解码、上采样并增加真实感。这些方法无法重建细节丰富的头发,难以处理卷发,或者仅限于处理少数几种发型。为克服这些局限,我们提出了DiffLocks,这是一个新颖的框架,能够直接从单张图像中详细重建多种多样的发型。首先,我们通过自动化创建迄今为止最大的合成头发数据集(包含4万种发型)来解决三维头发数据缺乏的问题。其次,我们利用该合成头发数据集学习一个以图像为条件的扩散变换器模型,从单张正面图像生成精确的三维发丝。通过使用预训练的图像骨干网络,尽管仅在合成数据上训练,我们的方法仍能泛化到自然场景图像。我们的扩散模型预测一张头皮纹理图,图中任意一点都包含单根发丝的潜在编码。这些编码可直接解码为三维发丝,无需后处理技术。表示单根发丝而非引导发丝,使变换器能够建模复杂发型的详细空间结构。凭借这一点,DiffLocks首次能够从单张图像中恢复高度卷曲的头发,例如非洲式发型。数据和代码可在https://radualexandru.github.io/difflocks/获取。

英文摘要:

We address the task of generating 3D hair geometry from a single image, which is challenging due to the diversity of hairstyles and the lack of paired image-to-3D hair data. Previous methods are primarily trained on synthetic data and cope with the limited amount of such data by using low-dimensional intermediate representations, such as guide strands and scalp-level embeddings, that require post-processing to decode, upsample, and add realism. These approaches fail to reconstruct detailed hair, struggle with curly hair, or are limited to handling only a few hairstyles. To overcome these limitations, we propose DiffLocks, a novel framework that enables detailed reconstruction of a wide variety of hairstyles directly from a single image. First, we address the lack of 3D hair data by automating the creation of the largest synthetic hair dataset to date, containing 40K hairstyles. Second, we leverage the synthetic hair dataset to learn an image-conditioned diffusion-transfomer model that generates accurate 3D strands from a single frontal image. By using a pretrained image backbone, our method generalizes to in-the-wild images despite being trained only on synthetic data. Our diffusion model predicts a scalp texture map in which any point in the map contains the latent code for an individual hair strand. These codes are directly decoded to 3D strands without post-processing techniques. Representing individual strands, instead of guide strands, enables the transformer to model the detailed spatial structure of complex hairstyles. With this, DiffLocks can recover highly curled hair, like afro hairstyles, from a single image for the first time. Data and code is available at https://radualexandru.github.io/difflocks/

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑