360°体积人像Avatar
360° Volumetric Portrait Avatar
AI总结：
本文提出3VP Avatar，通过模板化躯干、头部与表情跟踪及神经辐射场和变形场混合基，仅用单目旋转视频实现首个完整360°照片级真实感人像重建。
摘要：
我们提出360° Volumetric Portrait（3VP）Avatar，这是一种仅基于单目视频输入来重建人物360°照片级真实感人像Avatar的新颖方法。最先进的单目Avatar重建方法依赖于稳定的面部表情捕捉。然而，常用的基于3DMM的面部跟踪存在局限：侧面视角几乎无法捕捉，而且尤其会在背面视角失败，因为面部关键点或人体解析掩膜等所需输入缺失。这导致不完整的Avatar重建，只能覆盖前半球。与此不同，我们提出对躯干、头部和面部表情进行基于模板的跟踪，从而能够覆盖人物各个方向的外观。因此，给定人物在单个相机前旋转的序列，我们基于神经辐射场训练一种神经体积表示。构建该表示的一个关键挑战是对外观变化进行建模，尤其是嘴部区域（即嘴唇和牙齿）的变化。为此，我们提出一种基于变形场的混合基，使我们能够在不同外观状态之间进行插值。我们在采集的真实世界数据上评估了该方法，并与最先进的单目重建方法进行比较。与这些方法相比，我们的方法是首个能够重建完整360° Avatar的单目技术。
英文摘要：
We propose 360° Volumetric Portrait (3VP) Avatar, a novel method for reconstructing 360° photo-realistic portrait avatars of human subjects solely based on monocular video inputs. State-of-the-art monocular avatar reconstruction methods rely on stable facial performance capturing. However, the common usage of 3DMM-based facial tracking has its limits; side-views can hardly be captured and it fails, especially, for back-views, as required inputs like facial landmarks or human parsing masks are missing. This results in incomplete avatar reconstructions that only cover the frontal hemisphere. In contrast to this, we propose a template-based tracking of the torso, head and facial expressions which allows us to cover the appearance of a human subject from all sides. Thus, given a sequence of a subject that is rotating in front of a single camera, we train a neural volumetric representation based on neural radiance fields. A key challenge to construct this representation is the modeling of appearance changes, especially, in the mouth region (i.e., lips and teeth). We, therefore, propose a deformation-field-based blend basis which allows us to interpolate between different appearance states. We evaluate our approach on captured real-world data and compare against state-of-the-art monocular reconstruction methods. In contrast to those, our method is the first monocular technique that reconstructs an entire 360° avatar.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
