arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2606.11180cs.CV

Lip Forcing: 用于实时唇部同步的少步自回归扩散

Lip Forcing: Few-Step Autoregressive Diffusion for Real-time Lip Synchronization

KAIST AI(韩国科学技术院人工智能);AIPARK

Paul Hyunbin Cho, Jinhyuk Jang, SeokYoung Lee, Joungbin Lee, Siyoon Jin, Heeseong Shin, Jung Yi, Yunjin Park, Chulmin Park, Seungryong Kim

AI总结:

提出Lip Forcing,首个用于视频到视频唇部同步的自回归扩散方法,通过蒸馏14B教师模型为因果学生模型,仅需两步去噪即可实现实时同步,并引入同步窗口DMD、两步推理计划和SyncNet奖励。

摘要:

基于扩散的唇部同步模型实现了强大的视觉质量和音视频对齐,但全序列双向注意力和大量去噪步骤使其不适用于实时推理。我们提出了Lip Forcing,据我们所知,这是首个用于视频到视频(V2V)唇部同步的自回归扩散方法,它将一个14B音频条件双向视频扩散教师模型蒸馏为因果学生模型。在推理时,学生模型仅需两步去噪即可生成每个块,无需推理时的CFG,从而实现实时唇部同步。针对唇部同步的教师轨迹分析揭示了CFG保真度-同步权衡:无CFG预测偏向参考保真度,而CFG引导预测偏向中间轨迹带内的同步。Lip Forcing将这一发现转化为三个分析导出的组件:同步窗口DMD、两步推理计划和基于SyncNet的奖励。我们在两个学生尺度上验证了Lip Forcing,两者均从14B教师模型蒸馏而来。1.3B学生模型以31 FPS的速度进入实时流式处理,比同尺度的双向模型快17.6倍。14B学生模型是V2V唇部同步中报道的最大扩散模型,在参考保真度相当的情况下,运行速度比其教师模型快39.8倍。两种尺度的首帧时间均为亚毫秒级,远低于所有扩散基线。

英文摘要:

Diffusion-based lip synchronization models achieve strong visual quality and audio-visual alignment, but full-sequence bidirectional attention and many denoising steps make them impractical for real-time inference. We present Lip Forcing, to our knowledge the first autoregressive diffusion method for video-to-video (V2V) lip synchronization, which distills a 14B audio-conditioned bidirectional video diffusion teacher into causal students. At inference, the students generate each chunk in only two denoising steps without inference-time CFG, enabling real-time lip synchronization. A lip-sync-specific teacher-trajectory analysis reveals a CFG fidelity-sync tradeoff: no-CFG predictions favor reference fidelity, whereas CFG-guided predictions favor synchronization within a mid-trajectory band. Lip Forcing translates this finding into three analysis-derived components: Sync-Window DMD, a two-step inference schedule, and a SyncNet-based reward. We validate Lip Forcing at two student scales, both distilled from the 14B teacher. The 1.3B student crosses into real-time streaming at 31 FPS, $17.6\times$ faster than its same-scale bidirectional model. The 14B student, the largest diffusion model reported for V2V lip synchronization, runs $39.8\times$ faster than its teacher at comparable reference fidelity. Time-to-first-frame is sub-millisecond at both scales, far below every diffusion baseline.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑