Semantic-Human：基于人体解析的单目视频人体神经渲染
Semantic-Human: Neural Rendering of Humans from Monocular Video with Human Parsing
AI总结：
针对现有神经渲染忽略人体解析、运动场泛化性差的问题，提出Semantic-Human方法，扩展NeRF联合编码语义等信息，结合SMPL约束，在ZJU-MoCap数据集上取得优异效果并支持多种应用。
摘要：
人体神经渲染是一个具有重要研究意义的课题。然而，以往的研究大多聚焦于实现照片级真实感细节，忽略了对人体解析的探索。此外，经典的语义研究工作在高效呈现复杂运动下的精细结果方面均存在局限。人体解析与辐射重建存在内在关联，因为相似的外观和几何结构通常对应相似的语义部位。再者，以往研究通常设计一个从观测空间映射到规范空间的运动场，但该运动场往往存在欠拟合或过拟合问题，导致泛化能力有限。本文提出了Semantic-Human这一新型方法，可在人体神经渲染中同时实现照片级真实感细节和视角一致的人体解析。具体而言，我们扩展了神经辐射场（NeRF），联合编码语义、外观和几何结构，以在含噪伪标签监督下获得准确的2D语义标签。借助NeRF固有的一致性和平滑性特性，Semantic-Human在连续视角和新视角下均能实现一致的人体解析。我们还引入了源自SMPL表面的运动场约束，以及对恢复的体积几何结构的正则化。我们使用ZJU-MoCap数据集对模型进行了评估，获得的极具竞争力的结果证明了所提出的Semantic-Human的有效性。我们还展示了多种引人注目的应用，包括标签去噪、标签合成和图像编辑，并通过实验验证了其优越特性。
英文摘要：
The neural rendering of humans is a topic of great research significance. However, previous works mostly focus on achieving photorealistic details, neglecting the exploration of human parsing. Additionally, classical semantic work are all limited in their ability to efficiently represent fine results in complex motions. Human parsing is inherently related to radiance reconstruction, as similar appearance and geometry often correspond to similar semantic part. Furthermore, previous works often design a motion field that maps from the observation space to the canonical space, while it tends to exhibit either underfitting or overfitting, resulting in limited generalization. In this paper, we present Semantic-Human, a novel method that achieves both photorealistic details and viewpoint-consistent human parsing for the neural rendering of humans. Specifically, we extend neural radiance fields (NeRF) to jointly encode semantics, appearance and geometry to achieve accurate 2D semantic labels using noisy pseudo-label supervision. Leveraging the inherent consistency and smoothness properties of NeRF, Semantic-Human achieves consistent human parsing in both continuous and novel views. We also introduce constraints derived from the SMPL surface for the motion field and regularization for the recovered volumetric geometry. We have evaluated the model using the ZJU-MoCap dataset, and the obtained highly competitive results demonstrate the effectiveness of our proposed Semantic-Human. We also showcase various compelling applications, including label denoising, label synthesis and image editing, and empirically validate its advantageous properties.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
