GaussianTalker：基于音频驱动3D Gaussian Splatting的实时高保真说话头部合成
GaussianTalker: Real-Time High-Fidelity Talking Head Synthesis with Audio-Driven 3D Gaussian Splatting
AI总结：
提出GaussianTalker框架，利用3D Gaussian Splatting和空间-音频注意力模块实现实时高保真、姿态可控的说话头部合成，渲染速度达120 FPS。
摘要：
我们提出了GaussianTalker，一个用于实时生成姿态可控说话头部的新颖框架。它利用了3D Gaussian Splatting（3DGS）的快速渲染能力，同时解决了直接用语音音频控制3DGS的挑战。GaussianTalker构建了头部的规范3DGS表示，并根据音频同步对其进行形变。一个关键洞察是将3D高斯属性编码在共享的隐式特征表示中，在此处将其与音频特征合并以操纵每个高斯属性。这种设计利用了空间感知特征，并强制相邻点之间进行交互。然后，这些特征被馈送到空间-音频注意力模块，该模块预测每个高斯属性的逐帧偏移量。与以往用于操纵大量高斯及其精细参数的拼接或乘法方法相比，它更加稳定。实验结果展示了GaussianTalker在面部保真度、唇部同步精度和渲染速度上相比以往方法的优越性。具体而言，GaussianTalker实现了高达120 FPS的显著渲染速度，超越了以往的基准。我们的代码已在https://github.com/KU-CVLAB/GaussianTalker/ 开源。
英文摘要：
We propose GaussianTalker, a novel framework for real-time generation of pose-controllable talking heads. It leverages the fast rendering capabilities of 3D Gaussian Splatting (3DGS) while addressing the challenges of directly controlling 3DGS with speech audio. GaussianTalker constructs a canonical 3DGS representation of the head and deforms it in sync with the audio. A key insight is to encode the 3D Gaussian attributes into a shared implicit feature representation, where it is merged with audio features to manipulate each Gaussian attribute. This design exploits the spatial-aware features and enforces interactions between neighboring points. The feature embeddings are then fed to a spatial-audio attention module, which predicts frame-wise offsets for the attributes of each Gaussian. It is more stable than previous concatenation or multiplication approaches for manipulating the numerous Gaussians and their intricate parameters. Experimental results showcase GaussianTalker's superiority in facial fidelity, lip synchronization accuracy, and rendering speed compared to previous methods. Specifically, GaussianTalker achieves a remarkable rendering speed up to 120 FPS, surpassing previous benchmarks. Our code is made available at https://github.com/KU-CVLAB/GaussianTalker/ .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
