发表机构
China Telecom Artificial Intelligence Technology (Beijing) Co., Ltd（中国电信人工智能技术（北京）有限公司）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文提出Ripple系统，通过跨模态循环记忆机制及三阶段训练方案，实现480P下约28 FPS的实时流音频-视频生成，性能优于现有方法。
AI 中文摘要
音频-视频生成模型虽能生成高质量内容，但存在高延迟问题，无法适配实时应用场景。尽管已有若干流音频-视频生成方法被提出，但其仍存在成本高、无法支持长文本生成的缺陷。为解决该问题，本文提出Ripple，这是一种带有跨模态循环记忆机制的实时联合音频-视频生成系统。为在保留长期上下文的同时实现高效流推理，Ripple将固定长度滑动窗口注意力与针对音频、视频上下文进行持续总结的模态特定记忆状态相结合，还引入跨模态记忆交互以增强音视频同步性。为有效学习该记忆增强模型，本文设计了三阶段训练方案：（1）将双向音频-视频教师适配为带有模拟记忆的分块因果注意力；（2）通过端到端蒸馏优化记忆构建与交互流程；（3）应用专为流音频-视频生成定制的在线强化后训练。最终，Ripple在480P分辨率下实现约28 FPS，比教师模型快一倍以上，同时具备连贯的长文本生成能力。在短视频和长视频基准上开展的大量实验表明，本文方法相较于现有离线和在线联合音频-视频生成方法具备更优性能。
英文摘要
Audio-video generative models achieve impressive quality but suffer from high latency, making them unsuitable for real-time applications. Although several streaming audio-video generation methods have been proposed, they remain costly and fail to support long-form generation. To address this, we propose \textbf{Ripple}, a real-time joint audio-video generation system with a cross-modal recurrent memory mechanism. To enable efficient streaming inference while preserving long-term context, Ripple combines a fixed-length sliding-window attention with modality-specific memory states that continuously summarize audio and video context. Cross-modal memory interaction is further introduced to enhance audio-visual synchronization. To learn this memory-augmented model effectively, we devise a three-stage training recipe: (1) adapting a bidirectional audio-video teacher to block-wise causal attention with simulated memory, (2) optimizing the memory construction and interaction pipeline through end-to-end distillation, and (3) applying online reinforcement post-training tailored for streaming audio-video generation. As a result, Ripple achieves ~28 FPS at 480P resolution, over faster than the teacher, while capable of coherent long-form generation. Extensive experiments on both short-video and long-video benchmarks demonstrate our superior performance over existing offline and online joint audio-video generation methods.
