DyStream: 通过基于流匹配的自回归模型实现流式双人谈话头生成
DyStream: Streaming Dyadic Talking Heads Generation via Flow Matching-based Autoregressive Model
AI总结：
DyStream通过基于流匹配的自回归模型实现双人谈话头视频的实时生成，有效降低延迟并提升唇同步质量。
摘要：
生成逼真的双人谈话头视频需要超低延迟。现有基于分块的方法需要完整的非因果上下文窗口，导致显著延迟。这种高延迟严重阻碍了真实听众所需的即时非语言反馈。为了解决这个问题，我们提出了DyStream，一种基于流匹配的自回归模型，能够从说话者和听众的音频实时生成视频。我们的方法包含两个关键设计：（1）我们采用了一种友好的流式自回归框架，带有流匹配头的概率建模；（2）我们提出了一种增强的因果编码器，通过一个前瞻模块来整合短未来的上下文（例如60毫秒），以在保持低延迟的同时提高质量。我们的分析表明，这种简单而有效的办法显著超越了替代的因果策略，包括蒸馏和生成编码器。广泛的实验表明，DyStream能够在每帧34毫秒内生成视频，保证整个系统的延迟保持在100毫秒以下。此外，它在HDTF上实现了最先进的唇同步质量，离线和在线LipSync Confidence分数分别为8.13和7.61。该模型、权重和代码已公开。
英文摘要：
Generating realistic, dyadic talking head video requires ultra-low latency. Existing chunk-based methods require full non-causal context windows, introducing significant delays. This high latency critically prevents the immediate, non-verbal feedback required for a realistic listener. To address this, we present DyStream, a flow matching-based autoregressive model that could generate video in real-time from both speaker and listener audio. Our method contains two key designs: (1) we adopt a stream-friendly autoregressive framework with flow-matching heads for probabilistic modeling, and (2) We propose a causal encoder enhanced by a lookahead module to incorporate short future context (e.g., 60 ms) to improve quality while maintaining low latency. Our analysis shows this simple-and-effective method significantly surpass alternative causal strategies, including distillation and generative encoder. Extensive experiments show that DyStream could generate video within 34 ms per frame, guaranteeing the entire system latency remains under 100 ms. Besides, it achieves state-of-the-art lip-sync quality, with offline and online LipSync Confidence scores of 8.13 and 7.61 on HDTF, respectively. The model, weights and codes are available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
