面向可重光照与可交互面部渲染的即时面部高斯转换器
Instant Facial Gaussians Translator for Relightable and Interactable Facial Rendering
AI总结：
本文提出GauFace高斯表示和TransGS扩散Transformer，实现基于物理面部资产的即时转换与实时可重光照、可交互渲染，并在移动端和VR等平台验证其高效性与高质量。
摘要：
我们提出GauFace，一种新颖的高斯泼溅表示，专为基于物理的面部资产的高效动画与渲染而定制。借助强几何先验和约束优化，GauFace确保了整洁且结构化的高斯表示，在骁龙8 Gen 2移动平台上以30fps@1440p实现高保真和实时面部交互。随后，我们引入TransGS，一种扩散Transformer，能够即时将基于物理的面部资产转换为相应的GauFace表示。具体而言，我们采用基于分块（patch-based）的流水线来有效处理大量高斯。我们还引入了一种新颖的像素对齐采样方案，并结合UV位置编码，以确保由TransGS生成的GauFace资产具有足够的吞吐量和渲染质量。一旦训练完成，TransGS可以即时将带光照条件的面部资产转换为GauFace表示。凭借丰富的条件模态，它还实现了类似传统CG流程的编辑和动画能力。我们进行了广泛的评估和用户研究，与传统离线和在线渲染器以及近期神经渲染方法进行比较，证明了我们的方法在面部资产渲染方面的优越性能。我们还展示了使用TransGS方法和GauFace表示的面部资产在PC、手机甚至VR头显等多种平台上的多样化沉浸式应用。
英文摘要：
We propose GauFace, a novel Gaussian Splatting representation, tailored for efficient animation and rendering of physically-based facial assets. Leveraging strong geometric priors and constrained optimization, GauFace ensures a neat and structured Gaussian representation, delivering high fidelity and real-time facial interaction of 30fps@1440p on a Snapdragon 8 Gen 2 mobile platform. Then, we introduce TransGS, a diffusion transformer that instantly translates physically-based facial assets into the corresponding GauFace representations. Specifically, we adopt a patch-based pipeline to handle the vast number of Gaussians effectively. We also introduce a novel pixel-aligned sampling scheme with UV positional encoding to ensure the throughput and rendering quality of GauFace assets generated by our TransGS. Once trained, TransGS can instantly translate facial assets with lighting conditions to GauFace representation, With the rich conditioning modalities, it also enables editing and animation capabilities reminiscent of traditional CG pipelines. We conduct extensive evaluations and user studies, compared to traditional offline and online renderers, as well as recent neural rendering methods, which demonstrate the superior performance of our approach for facial asset rendering. We also showcase diverse immersive applications of facial assets using our TransGS approach and GauFace representation, across various platforms like PCs, phones and even VR headsets.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
