面向沉浸式以人为主体的体积视频的鲁棒双高斯溅射方法
Robust Dual Gaussian Splatting for Immersive Human-centric Volumetric Videos
AI总结：
针对现有体积视频工作流人工干预多、资源体积大的问题，提出DualGS双高斯方法，通过运动与外观解耦、由粗到精训练及高效压缩，实现高压缩比的实时高保真人体表演回放，可支撑VR沉浸式自由观看体验。
摘要：
体积视频是视觉媒体领域的一项变革性进展，它使用户能够自由导航沉浸式虚拟体验，缩小了数字世界与现实世界之间的差距。然而，现有工作流程中需要大量人工干预来稳定网格序列，且生成的资源体积过大，这阻碍了其更广泛的应用。在本文中，我们提出了一种基于高斯的新方法，名为DualGS，可实现复杂人体表演的实时高保真回放，并具备出色的压缩比。DualGS的核心思路是利用对应的皮肤高斯和关节高斯分别表示运动和外观。这种显式解耦能够显著减少运动冗余并增强时间一致性。我们首先初始化DualGS，并在第一帧将皮肤高斯锚定到关节高斯上。随后，我们采用由粗到精的训练策略逐帧进行人体表演建模，该策略包括用于整体运动预测的粗对齐阶段，以及用于鲁棒跟踪和高保真渲染的细粒度优化阶段。为了将体积视频无缝集成到VR环境中，我们使用熵编码高效压缩运动信息，并结合持久码本使用编解码器压缩外观信息。我们的方法实现了高达120倍的压缩比，每帧仅需约350KB的存储空间。我们通过VR头显上的照片级真实感自由视角体验验证了该表示方法的有效性，用户能够沉浸式观看音乐家的表演，感受表演者指尖下的音符节奏。
英文摘要：
Volumetric video represents a transformative advancement in visual media, enabling users to freely navigate immersive virtual experiences and narrowing the gap between digital and real worlds. However, the need for extensive manual intervention to stabilize mesh sequences and the generation of excessively large assets in existing workflows impedes broader adoption. In this paper, we present a novel Gaussian-based approach, dubbed \textit{DualGS}, for real-time and high-fidelity playback of complex human performance with excellent compression ratios. Our key idea in DualGS is to separately represent motion and appearance using the corresponding skin and joint Gaussians. Such an explicit disentanglement can significantly reduce motion redundancy and enhance temporal coherence. We begin by initializing the DualGS and anchoring skin Gaussians to joint Gaussians at the first frame. Subsequently, we employ a coarse-to-fine training strategy for frame-by-frame human performance modeling. It includes a coarse alignment phase for overall motion prediction as well as a fine-grained optimization for robust tracking and high-fidelity rendering. To integrate volumetric video seamlessly into VR environments, we efficiently compress motion using entropy encoding and appearance using codec compression coupled with a persistent codebook. Our approach achieves a compression ratio of up to 120 times, only requiring approximately 350KB of storage per frame. We demonstrate the efficacy of our representation through photo-realistic, free-view experiences on VR headsets, enabling users to immersively watch musicians in performance and feel the rhythm of the notes at the performers' fingertips.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
