MonoHuman：基于单目视频的可动画化人体神经场
MonoHuman: Animatable Human Neural Field from Monocular Video
AI总结：
提出MonoHuman框架，通过共享双向形变模块和正向对应搜索模块，利用双向约束和关键帧信息，实现了在任意新颖姿态下渲染视角一致且高保真的可动画化人体化身。
摘要：
对具有自由视角控制的虚拟化身进行动画化，对于虚拟现实和数字娱乐等各种应用至关重要。先前的研究曾尝试利用神经辐射场（NeRF）的表示能力从单目视频中重建人体。最近的工作提出将形变网络移植到NeRF中，以进一步对人体神经场的动态进行建模，从而生成生动的人体运动。然而，此类流程要么依赖于依赖于姿态的表示，要么由于逐帧独立优化而缺乏运动连贯性，难以真实地泛化到未见过的姿态序列。在本文中，我们提出了一个新颖的框架MonoHuman，它能够在任意新颖姿态下稳健地渲染视角一致且高保真的化身。我们的关键见解是利用双向约束对形变场进行建模，并显式利用现成的关键帧信息来推理特征相关性，以获得连贯的结果。具体而言，我们首先提出了一个共享双向形变模块，通过将正向和反向形变对应关系解耦为共享的骨骼运动权重和独立的非刚性运动，创建一个不依赖姿态且可泛化的形变场。然后，我们设计了一个正向对应搜索模块，该模块查询关键帧的对应特征以指导渲染网络。因此，即使在具有挑战性的新颖姿态设置下，渲染结果也具有多视角一致性和高保真度。大量实验证明了我们提出的MonoHuman优于现有最先进的方法。
英文摘要：
Animating virtual avatars with free-view control is crucial for various applications like virtual reality and digital entertainment. Previous studies have attempted to utilize the representation power of the neural radiance field (NeRF) to reconstruct the human body from monocular videos. Recent works propose to graft a deformation network into the NeRF to further model the dynamics of the human neural field for animating vivid human motions. However, such pipelines either rely on pose-dependent representations or fall short of motion coherency due to frame-independent optimization, making it difficult to generalize to unseen pose sequences realistically. In this paper, we propose a novel framework MonoHuman, which robustly renders view-consistent and high-fidelity avatars under arbitrary novel poses. Our key insight is to model the deformation field with bi-directional constraints and explicitly leverage the off-the-peg keyframe information to reason the feature correlations for coherent results. Specifically, we first propose a Shared Bidirectional Deformation module, which creates a pose-independent generalizable deformation field by disentangling backward and forward deformation correspondences into shared skeletal motion weight and separate non-rigid motions. Then, we devise a Forward Correspondence Search module, which queries the correspondence feature of keyframes to guide the rendering network. The rendered results are thus multi-view consistent with high fidelity, even under challenging novel pose settings. Extensive experiments demonstrate the superiority of our proposed MonoHuman over state-of-the-art methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
