arXiv 2404.084492025-02-20cs.CV

OccGaussian：用于遮挡人体渲染的 3D Gaussian Splatting

OccGaussian: 3D Gaussian Splatting for Occluded Human Rendering

Jingrui Ye, Zongkai Zhang, Yujiao Jiang, Qingmin Liao, Wenming Yang, Zongqing Lu

备注：: We have decided to withdraw this paper because the results require further verification or additional experimental data. We plan to resubmit an updated version once the necessary work is completed

阅读 PDF

arXiv 页面 ↗HTML 原文 ↗

AI总结：

针对单目视频中人体被遮挡导致的渲染难题，本文提出基于 3D Gaussian Splatting 的 OccGaussian，通过遮挡特征查询与遮挡感知损失实现高质量快速渲染。

摘要：

从单目视频渲染动态 3D 人体对于虚拟现实和数字娱乐等多种应用至关重要。大多数方法假设人物处于无遮挡场景中，但在现实场景里，各种物体可能造成身体部位被遮挡。先前方法利用 NeRF 进行表面渲染以恢复被遮挡区域，但它需要一天以上的训练时间和数秒的渲染时间，无法满足实时交互应用的要求。为解决这些问题，我们提出了基于 3D Gaussian Splatting 的 OccGaussian，它可在 6 分钟内完成训练，并能在存在遮挡输入的情况下生成高达 160 FPS 的高质量人体渲染结果。OccGaussian 在规范空间中初始化 3D 高斯分布，并在被遮挡区域执行遮挡特征查询，提取聚合的像素对齐特征以补偿缺失信息。随后，我们使用 Gaussian Feature MLP 进一步处理这些特征，并结合遮挡感知损失函数，以更好地感知被遮挡区域。在模拟遮挡和真实世界遮挡中的大量实验表明，与最先进方法相比，我们的方法取得了相当甚至更优的性能。同时，我们将训练和推理速度分别提升了 250 倍和 800 倍。我们的代码将面向研究用途开放。

英文摘要：

Rendering dynamic 3D human from monocular videos is crucial for various applications such as virtual reality and digital entertainment. Most methods assume the people is in an unobstructed scene, while various objects may cause the occlusion of body parts in real-life scenarios. Previous method utilizing NeRF for surface rendering to recover the occluded areas, but it requiring more than one day to train and several seconds to render, failing to meet the requirements of real-time interactive applications. To address these issues, we propose OccGaussian based on 3D Gaussian Splatting, which can be trained within 6 minutes and produces high-quality human renderings up to 160 FPS with occluded input. OccGaussian initializes 3D Gaussian distributions in the canonical space, and we perform occlusion feature query at occluded regions, the aggregated pixel-align feature is extracted to compensate for the missing information. Then we use Gaussian Feature MLP to further process the feature along with the occlusion-aware loss functions to better perceive the occluded area. Extensive experiments both in simulated and real-world occlusions, demonstrate that our method achieves comparable or even superior performance compared to the state-of-the-art method. And we improving training and inference speeds by 250x and 800x, respectively. Our code will be available for research purposes.

机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。