HumanSplatHMR: 闭合人体网格恢复与高斯点绘肖像之间的循环
HumanSplatHMR: Closing the Loop Between Human Mesh Recovery and Gaussian Splatting Avatar
AI总结：
本文提出HumanSplatHMR方法，通过闭合几何姿态估计与可微渲染之间的循环，改进人体姿态恢复和高斯点绘肖像的生成，提升在新视角和新姿态下的渲染质量。
摘要：
从视频中准确恢复人体姿态和外观是场景重建的关键组成部分，应用于动作捕捉、动作预测、虚拟现实和数字孪生等领域。尽管对从视频中构建逼真人类肖像已有大量研究，本文证明现有方法无法准确恢复人类的3D几何结构。基于ViT的方法不一致可靠且可能过度拟合2D视角，而基于NeRF和高斯点绘的肖像将姿态和外观分开，限制了对新姿态的渲染泛化能力。为解决这些问题，本文提出HumanSplatHMR，一种联合优化框架，通过同时优化3D人体姿态并学习高保真的肖像，以实现新视角和新姿态的合成。我们的关键见解是闭合几何姿态估计与可微渲染之间的循环。不同于以往依赖运动捕捉系统或离线优化获得的准确人体姿态的人形肖像方法，在野外场景中不实用，我们的方法仅使用最先进的姿态估计器得到的人体网格估计，以更好地反映现实情况。因此，不同于将人体姿态仅作为变形先验使用，HumanSplatHMR通过可微渲染将光度、分割和深度损失反向传播到姿态参数和全局位置。这种耦合在时间上优化全局3D姿态，提高精度和对齐性，同时产生更高质量的新视角渲染。实验显示，与省略图像级优化的姿态恢复基线和将姿态估计与肖像重建解耦的肖像基线相比，有持续的改进。
英文摘要：
Accurately recovering human pose and appearance from video is an essential component of scene reconstruction, with applications to motion capture, motion prediction, virtual reality, and digital twinning. Despite significant interest in building realistic human avatars from video, this paper demonstrates that existing methods do not accurately recover the 3D geometry of humans. ViT-based approaches are not consistently reliable and can overfit to 2D views, while NeRF- and Gaussian Splatting-based avatars treat pose and appearance separately, limiting rendering generalization to new poses. To resolve these shortcomings, this paper proposes HumanSplatHMR, a joint optimization framework that refines 3D human poses while simultaneously learning a high-fidelity avatar for novel-view and novel-pose synthesis. Our key insight is to close the loop between geometric pose estimation and differentiable rendering. Unlike prior human avatar methods that rely on accurate human pose obtained through motion capture systems or offline refinement, which are impractical in in-the-wild scenarios, our approach uses only human mesh estimates from a state-of-the-art human pose estimator to better reflect real-world conditions. Therefore, instead of using the human pose only as a deformation prior, HumanSplatHMR backpropagates photometric, segmentation, and depth losses through a differentiable renderer to the pose parameters and global position. This coupling refines the global 3D pose over time, improving accuracy and alignment while producing better renderings from novel views. Experiments show consistent improvements over pose recovery baselines that omit image-level refinement and avatar baselines that decouple pose estimation from avatar reconstruction.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
