TempoSyncDiff: 压缩时间一致扩散用于低延迟音频驱动说话头生成
TempoSyncDiff: Distilled Temporally-Consistent Diffusion for Low-Latency Audio-Driven Talking Head Generation
AI总结：
TempoSyncDiff通过压缩扩散模型实现低延迟音频驱动说话头生成，结合教师-学生压缩和时间正则化以提高生成稳定性。
摘要：
扩散模型最近在逼真人类合成方面取得了进展，尽管实际的说话头生成（THG）仍受高推理延迟、时间不稳定（如闪烁和身份漂移）以及在挑战性语音条件下不完美的音频-视觉对齐的限制。本文介绍了TempoSyncDiff，一种参考条件化的潜在扩散框架，探索了少量步骤推理以实现高效的音频驱动说话头生成。该方法采用教师-学生压缩公式，在标准噪声预测目标训练的扩散教师指导下，一个轻量级的学生去噪器能够以显著更少的推理步骤运行，以提高生成稳定性。该框架结合了身份锚定和时间正则化，旨在在合成过程中减轻身份漂移和帧间闪烁，同时基于音素的音频条件化提供了粗略的唇部运动控制。在LRs3数据集上的实验报告了去噪阶段组件级指标，相对于VAE重建，并初步的延迟特征，包括仅CPU和边缘计算测量以及边缘部署的可行性估计。结果表明，压缩的扩散模型可以保留更强教师的大部分重建行为，同时实现显著更低的推理延迟。本研究被定位为在受限计算条件下实现实用扩散说话头生成的初步步骤。
英文摘要：
Diffusion models have recently advanced photorealistic human synthesis, although practical talking-head generation (THG) remains constrained by high inference latency, temporal instability such as flicker and identity drift, and imperfect audio-visual alignment under challenging speech conditions. This paper introduces TempoSyncDiff, a reference-conditioned latent diffusion framework that explores few-step inference for efficient audio-driven talking-head generation. The approach adopts a teacher-student distillation formulation in which a diffusion teacher trained with a standard noise prediction objective guides a lightweight student denoiser capable of operating with significantly fewer inference steps to improve generation stability. The framework incorporates identity anchoring and temporal regularization designed to mitigate identity drift and frame-to-frame flicker during synthesis, while viseme-based audio conditioning provides coarse lip motion control. Experiments on the LRS3 dataset report denoising-stage component-level metrics relative to VAE reconstructions and preliminary latency characterization, including CPU-only and edge computing measurements and feasibility estimates for edge deployment. The results suggest that distilled diffusion models can retain much of the reconstruction behaviour of a stronger teacher while enabling substantially lower latency inference. The study is positioned as an initial step toward practical diffusion-based talking-head generation under constrained computational settings. GitHub: https://mazumdarsoumya.github.io/TempoSyncDiff
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
