使用混合3D表示学习解耦的虚拟人
Learning Disentangled Avatars with Hybrid 3D Representations
AI总结：
提出DELTA框架，采用混合显式-隐式3D表示（网格参数化模型与神经辐射场）从单目视频解耦建模人体、服装与面部头发，实现无3D监督的解耦重建及虚拟试衣与发型迁移。
摘要：
为了学习可动画且逼真的人体虚拟人，人们已经付出了巨大的努力。为此，显式和隐式3D表示都被广泛研究，以实现对整个人体（例如身体、服装、面部和头发）的整体建模和捕捉，但就表示效能而言，这两种表示都不是最优选择，因为人体虚拟人的不同部分有不同的建模需求。例如，网格通常不适合用于建模服装和头发。受此启发，我们提出了Disentangled Avatars (DELTA)，它使用混合的显式-隐式3D表示来对人体进行建模。DELTA以单目RGB视频作为输入，并生成具有独立身体和服装/头发层的人体虚拟人。具体来说，我们展示了DELTA的两个重要应用。第一个应用中，我们考虑人体与服装的解耦，第二个应用中，我们解耦面部和头发。为此，DELTA使用基于网格的显式参数化3D模型来表示身体或面部，并使用隐式神经辐射场来表示服装或头发。为了实现这一点，我们设计了一个端到端的可微渲染器，将网格集成到体积渲染中，使DELTA能够直接从单目视频中学习，而无需任何3D监督。最后，我们展示了如何轻松组合这两个应用来建模全身虚拟人，使得头发、面部、身体和服装可以完全解耦并同时联合渲染。这种解耦使得头发和服装能够转移到任意身体形状上。我们通过展示DELTA在解耦重建、虚拟试衣和发型迁移上的优异性能，从经验上验证了其解耦的有效性。为了促进未来的研究，我们还发布了一个开源流水线用于混合人体虚拟人建模的研究。
英文摘要：
Tremendous efforts have been made to learn animatable and photorealistic human avatars. Towards this end, both explicit and implicit 3D representations are heavily studied for a holistic modeling and capture of the whole human (e.g., body, clothing, face and hair), but neither representation is an optimal choice in terms of representation efficacy since different parts of the human avatar have different modeling desiderata. For example, meshes are generally not suitable for modeling clothing and hair. Motivated by this, we present Disentangled Avatars~(DELTA), which models humans with hybrid explicit-implicit 3D representations. DELTA takes a monocular RGB video as input, and produces a human avatar with separate body and clothing/hair layers. Specifically, we demonstrate two important applications for DELTA. For the first one, we consider the disentanglement of the human body and clothing and in the second, we disentangle the face and hair. To do so, DELTA represents the body or face with an explicit mesh-based parametric 3D model and the clothing or hair with an implicit neural radiance field. To make this possible, we design an end-to-end differentiable renderer that integrates meshes into volumetric rendering, enabling DELTA to learn directly from monocular videos without any 3D supervision. Finally, we show that how these two applications can be easily combined to model full-body avatars, such that the hair, face, body and clothing can be fully disentangled yet jointly rendered. Such a disentanglement enables hair and clothing transfer to arbitrary body shapes. We empirically validate the effectiveness of DELTA's disentanglement by demonstrating its promising performance on disentangled reconstruction, virtual clothing try-on and hairstyle transfer. To facilitate future research, we also release an open-sourced pipeline for the study of hybrid human avatar modeling.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
