KSDiff: 关键帧增强型语音感知双路径扩散模型用于面部动画
KSDiff: Keyframe-Augmented Speech-Aware Dual-Path Diffusion for Facial Animation
AI总结：
本文提出KSDiff框架，通过双路径语音编码器和关键帧建立学习模块，提升语音驱动面部动画的唇同步和头部姿态自然度。
摘要：
音频驱动面部动画在多媒体应用中取得显著进展，扩散模型在说话人脸合成中展现出强大潜力。然而，现有方法将语音特征视为单一表示，未能捕捉其细粒度作用于不同面部运动，同时忽视了对动态强烈的关键帧建模的重要性。为解决这些问题，我们提出了KSDiff，一种关键帧增强型语音感知双路径扩散框架。具体而言，原始音频和转录文本通过双路径语音编码器处理，以分离与表情相关的和头部姿态相关的特征，同时自回归的关键帧建立学习模块预测最显著的运动帧。这些组件整合到双路径运动生成器中，以合成连贯且逼真的面部运动。在HDTF和VoxCeleb上的广泛实验表明，KSDiff在唇同步准确性和头部姿态自然度上均取得最佳性能。我们的结果强调了将语音解耦与关键帧感知扩散结合在说话人生成中的有效性。演示页面可在：https://kincin.github.io/KSDiff/。
英文摘要：
Audio-driven facial animation has made significant progress in multimedia applications, with diffusion models showing strong potential for talking-face synthesis. However, most existing works treat speech features as a monolithic representation and fail to capture their fine-grained roles in driving different facial motions, while also overlooking the importance of modeling keyframes with intense dynamics. To address these limitations, we propose KSDiff, a Keyframe-Augmented Speech-Aware Dual-Path Diffusion framework. Specifically, the raw audio and transcript are processed by a Dual-Path Speech Encoder (DPSE) to disentangle expression-related and head-pose-related features, while an autoregressive Keyframe Establishment Learning (KEL) module predicts the most salient motion frames. These components are integrated into a Dual-path Motion generator to synthesize coherent and realistic facial motions. Extensive experiments on HDTF and VoxCeleb demonstrate that KSDiff achieves state-of-the-art performance, with improvements in both lip synchronization accuracy and head-pose naturalness. Our results highlight the effectiveness of combining speech disentanglement with keyframe-aware diffusion for talking-head generation. The demo page is available at: https://kincin.github.io/KSDiff/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
