Ditto：用于可控实时说话头合成的运动空间扩散
Ditto: Motion-Space Diffusion for Controllable Realtime Talking Head Synthesis
AI总结：
Ditto通过运动空间扩散模型实现可控实时说话头合成，优化架构与训练策略以提升生成质量与实时性能。
摘要：
最近在扩散模型方面的进展使说话头合成具备了细腻的表情和生动的头部运动，但同时也导致了推理速度慢和生成结果控制不足的问题。为了解决这些问题，我们提出了Ditto，一种基于扩散的说话头框架，能够实现细粒度控制和实时推理。具体来说，我们利用现成的运动提取器，并设计了一个扩散变压器来在特定的运动空间中生成表示。我们优化了模型架构和训练策略，以解决生成运动表示时的问题，包括运动和身份之间的解耦不足以及表示内部的较大差异。此外，我们在建立运动表示与面部语义之间的映射时采用多样的条件信号，从而能够控制生成过程并纠正结果。此外，我们联合优化整体框架，以实现流式处理、实时推理和低首帧延迟，为如AI助手等交互应用提供关键功能。广泛的实验结果表明，Ditto生成有吸引力的说话头视频，并在可控性和实时性能方面表现出色。
英文摘要：
Recent advances in diffusion models have endowed talking head synthesis with subtle expressions and vivid head movements, but have also led to slow inference speed and insufficient control over generated results. To address these issues, we propose Ditto, a diffusion-based talking head framework that enables fine-grained controls and real-time inference. Specifically, we utilize an off-the-shelf motion extractor and devise a diffusion transformer to generate representations in a specific motion space. We optimize the model architecture and training strategy to address the issues in generating motion representations, including insufficient disentanglement between motion and identity, and large internal discrepancies within the representation. Besides, we employ diverse conditional signals while establishing a mapping between motion representation and facial semantics, enabling control over the generation process and correction of the results. Moreover, we jointly optimize the holistic framework to enable streaming processing, real-time inference, and low first-frame delay, offering functionalities crucial for interactive applications such as AI assistants. Extensive experimental results demonstrate that Ditto generates compelling talking head videos and exhibits superiority in both controllability and real-time performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
