GenLCA：从真实视频中生成全身体素的3D扩散模型
GenLCA: 3D Diffusion for Full-Body Avatars from In-the-Wild Videos
AI总结：
本文提出GenLCA，一种基于扩散的生成模型，能从文本和图像输入生成逼真全身体素并进行编辑。通过利用预训练的avatar重建模型作为可动画的3D分词器，解决视频中部分观测导致的模糊或透明问题，提升生成质量与泛化能力。
摘要：
我们提出了GenLCA，一种基于扩散的生成模型，用于从文本和图像输入生成和编辑逼真的全身体素。生成的体素忠实于输入，同时支持高保真的面部和全身体动画。核心思想是一种新的范式，使能够从部分可观测的2D数据中训练全身体素的3D扩散模型，使训练数据集可扩展到数百万个真实世界视频。这种可扩展性有助于GenLCA的逼真度和泛化能力。具体来说，我们通过重新利用预训练的feed-forward avatar重建模型作为可动画的3D分词器，将无结构的视频帧编码为结构化的3D分词。然而，大多数真实世界视频只提供身体部分的观测，导致3D分词中出现过多的模糊或透明伪影。为了解决这个问题，我们提出了一种新的可见性感知扩散训练策略，用可学习的分词替换无效区域，并仅在有效区域上计算损失。然后，我们在分词数据集上训练一个基于流的扩散模型，从而内生地保持由预训练的avatar重建模型提供的逼真度和可动画性。我们的方法有效利用大规模真实世界视频数据来训练扩散模型原生的3D模型。通过多样且高保真的生成和编辑结果，我们证明了该方法的有效性，优于现有解决方案。项目页面可在https://onethousandwu.com/GenLCA-Page上找到。
英文摘要：
We present GenLCA, a diffusion-based generative model for generating and editing photorealistic full-body avatars from text and image inputs. The generated avatars are faithful to the inputs, while supporting high-fidelity facial and full-body animations. The core idea is a novel paradigm that enables training a full-body 3D diffusion model from partially observable 2D data, allowing the training dataset to scale to millions of real-world videos. This scalability contributes to the superior photorealism and generalizability of GenLCA. Specifically, we scale up the dataset by repurposing a pretrained feed-forward avatar reconstruction model as an animatable 3D tokenizer, which encodes unstructured video frames into structured 3D tokens. However, most real-world videos only provide partial observations of body parts, resulting in excessive blurring or transparency artifacts in the 3D tokens. To address this, we propose a novel visibility-aware diffusion training strategy that replaces invalid regions with learnable tokens and computes losses only over valid regions. We then train a flow-based diffusion model on the token dataset, inherently maintaining the photorealism and animatability provided by the pretrained avatar reconstruction model. Our approach effectively enables the use of large-scale real-world video data to train a diffusion model natively in 3D. We demonstrate the efficacy of our method through diverse and high-fidelity generation and editing results, outperforming existing solutions by a large margin. The project page is available at https://onethousandwu.com/GenLCA-Page.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
