听觉有助于视觉吗？研究音频视频联合去噪用于视频生成
Does Hearing Help Seeing? Investigating Audio-Video Joint Denoising for Video Generation
AI总结：
本文提出通过音频视频联合去噪提升视频生成质量，验证了跨模态训练对构建更物理基础的世界模型的潜力。
摘要：
最近的音频视频生成系统表明，耦合模态不仅有助于音频视频同步，也对视频模态本身有益。我们提出一个根本性问题：在只关心视频质量的情况下，音频视频联合去噪训练是否能提高视频生成？为研究此问题，我们引入了一个参数高效且利用预训练文本到视频（T2V）和文本到音频（T2A）模块进行联合去噪的音频视频全DiT（AVFullDiT）架构。我们训练（i）一个使用AVFullDiT的T2AV模型，以及（ii）一个仅使用T2V的对照模型。我们的结果提供了首次系统证据表明，音频视频联合去噪能提供比同步更多的收益。我们在具有大尺寸和物体接触运动的挑战性子集上观察到一致的改进。我们假设预测音频作为一种特权信号，促使模型内部化视觉事件与其声音后果之间的因果关系（例如，碰撞×撞击声），这反过来会正则化视频动态。我们的发现表明，跨模态协同训练是开发更强大、更物理基础的世界模型的一种有前途的方法。代码和数据集将被公开发布。
英文摘要：
Recent audio-video generative systems suggest that coupling modalities benefits not only audio-video synchrony but also the video modality itself. We pose a fundamental question: Does audio-video joint denoising training improve video generation, even when we only care about video quality? To study this, we introduce a parameter-efficient Audio-Video Full DiT (AVFullDiT) architecture that leverages pre-trained text-to-video (T2V) and text-to-audio (T2A) modules for joint denoising. We train (i) a T2AV model with AVFullDiT and (ii) a T2V-only counterpart under identical settings. Our results provide the first systematic evidence that audio-video joint denoising can deliver more than synchrony. We observe consistent improvements on challenging subsets featuring large and object contact motions. We hypothesize that predicting audio acts as a privileged signal, encouraging the model to internalize causal relationships between visual events and their acoustic consequences (e.g., collision $\times$ impact sound), which in turn regularizes video dynamics. Our findings suggest that cross-modal co-training is a promising approach to developing stronger, more physically grounded world models. Code and dataset will be made publicly available.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
