arXiv 2408.096652024-11-19cs.CV

基于语义引导高斯泼溅的拓扑感知人体化身

Topology-aware Human Avatars with Semantically-guided Gaussian Splatting

School of Computer Science, Wuhan University（武汉大学计算机学院）；School of Software Engineering, Xi’an Jiao Tong University（西安交通大学软件学院）；AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能研究院）；Huazhong University of Science and Technology（华中科技大学）

Haoyu Zhao, Chen Yang, Hao Wang, Xingyue Zhao, Wei Shen

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

本文提出SG-GS方法，通过语义嵌入3D高斯、骨架驱动刚性与非刚性变形及语义标注器SHA，结合三种语义正则化策略，实现拓扑感知的高质量可动画人体化身重建。

摘要：

从单目视频中重建照片级真实且具有拓扑感知能力的可动画人体化身，在计算机视觉与图形学中仍具挑战性。近年来，使用3D高斯表示人体的方法相继出现，带来了更快的优化速度和实时渲染能力。然而，由于忽略了人体语义信息所表达的人体显式拓扑与内在结构的关键作用，这些方法难以实现人体化身的精细细节重建。为解决这一问题，我们提出SG-GS，利用嵌入语义的3D高斯、骨架驱动的刚性变形以及非刚性布料动力学变形来生成照片级真实的人体化身。随后，我们设计了一种语义人体标注器（SHA），利用SMPL的语义先验进行高效的身体部位语义标注。生成的标签用于指导高斯语义属性的优化。为捕捉人体显式拓扑结构，我们采用一个融合拓扑关联与几何关联的3D网络进行人体化身变形。我们还实施了三种关键策略以提升3D高斯的语义精度与渲染质量：带2D正则化的语义投影、语义引导的密度正则化以及带邻域一致性的语义感知正则化。大量实验表明，SG-GS在几何与外观重建性能上达到了当前最优水平。

英文摘要：

Reconstructing photo-realistic and topology-aware animatable human avatars from monocular videos remains challenging in computer vision and graphics. Recently, methods using 3D Gaussians to represent the human body have emerged, offering faster optimization and real-time rendering. However, due to ignoring the crucial role of human body semantic information which represents the explicit topological and intrinsic structure within human body, they fail to achieve fine-detail reconstruction of human avatars. To address this issue, we propose SG-GS, which uses semantics-embedded 3D Gaussians, skeleton-driven rigid deformation, and non-rigid cloth dynamics deformation to create photo-realistic human avatars. We then design a Semantic Human-Body Annotator (SHA) which utilizes SMPL's semantic prior for efficient body part semantic labeling. The generated labels are used to guide the optimization of semantic attributes of Gaussian. To capture the explicit topological structure of the human body, we employ a 3D network that integrates both topological and geometric associations for human avatar deformation. We further implement three key strategies to enhance the semantic accuracy of 3D Gaussians and rendering quality: semantic projection with 2D regularization, semantic-guided density regularization and semantic-aware regularization with neighborhood consistency. Extensive experiments demonstrate that SG-GS achieves state-of-the-art geometry and appearance reconstruction performance.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。

AI总结：

摘要：

英文摘要：

相关论文