OmniForcing：释放实时联合音频视觉生成
OmniForcing: Unleashing Real-time Joint Audio-Visual Generation
AI总结：
本文提出OmniForcing框架，通过因果蒸馏将双向扩散模型转化为高保真流式自回归生成器，解决双流架构中的训练不稳定问题，实现25FPS的实时生成性能。
摘要：
近期的联合音频视觉扩散模型在生成质量上表现优异，但因双向注意力依赖导致高延迟，限制了实时应用。我们提出OmniForcing，首个将离线双流双向扩散模型蒸馏为高保真流式自回归生成器的框架。然而，简单地将因果蒸馏应用于双流架构会导致严重训练不稳定，由于模态间的极端时间不对称性和生成的token稀疏性。我们通过引入非对称块因果对齐和零截断全局前缀，解决固有的信息密度差距。通过音频Sink Token机制和Identity RoPE约束，解决因果转换过程中由极端音频token稀疏性导致的梯度爆炸。最后，联合自蒸馏范式使模型在长滚动过程中动态自我纠正累积的跨模态误差。借助模态无关的滚动KV缓存推理方案，OmniForcing在单个GPU上实现约25FPS的流式生成性能，保持多模态同步和视觉质量与双向教师模型相当。Project Page: https://omniforcing.com
英文摘要：
Recent joint audio-visual diffusion models achieve remarkable generation quality but suffer from high latency due to their bidirectional attention dependencies, hindering real-time applications. We propose OmniForcing, the first framework to distill an offline, dual-stream bidirectional diffusion model into a high-fidelity streaming autoregressive generator. However, naively applying causal distillation to such dual-stream architectures triggers severe training instability, due to the extreme temporal asymmetry between modalities and the resulting token sparsity. We address the inherent information density gap by introducing an Asymmetric Block-Causal Alignment with a zero-truncation Global Prefix that prevents multi-modal synchronization drift. The gradient explosion caused by extreme audio token sparsity during the causal shift is further resolved through an Audio Sink Token mechanism equipped with an Identity RoPE constraint. Finally, a Joint Self-Forcing Distillation paradigm enables the model to dynamically self-correct cumulative cross-modal errors from exposure bias during long rollouts. Empowered by a modality-independent rolling KV-cache inference scheme, OmniForcing achieves state-of-the-art streaming generation at $\sim$25 FPS on a single GPU, maintaining multi-modal synchronization and visual quality on par with the bidirectional teacher.\textbf{Project Page:} \href{https://omniforcing.com}{https://omniforcing.com}
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
