AI 中文总结
针对现有基于条件扩散的视频到音频（V2A）生成方法需额外网络或强归纳偏置的问题，提出TD-V2A框架，利用时间差异（TD）增强视觉条件，提升了V2A生成质量。
AI 中文摘要
视频到音频（V2A）生成在图像到音频（I2A）生成的基础上引入了连续帧，这些帧为音频合成提供了必要的时间线索。然而，现有的基于条件扩散的V2A方法通常需要通过额外的视听监督、声学结构预测或大型多模态模型的推理来增强视觉条件，这需要额外的网络或强归纳偏置。受近期视觉表示学习进展的启发，我们提出TD-V2A，它利用时间差异（TD）作为区分V2A与I2A的关键表示，以最小的架构修改丰富视觉条件。我们首先在帧级和特征级研究TD，以确定TD补充视觉表示的最有效表示级别。基于这些发现，我们开发了分层持续学习策略和退火时间差异引导方法，分别在扩散训练和采样过程中逐步学习和利用TD信息。在基准数据集上的大量实验表明，通过我们提出的框架有效利用TD可显著提高端到端V2A生成质量，甚至优于对比视听预训练等专用V2A表示。
英文摘要
Video-to-audio (V2A) generation extends image-to-audio generation (I2A) by introducing consecutive frames that provide essential temporal cues for audio synthesis. However, existing conditional diffusion-based V2A methods typically enhance visual conditioning with additional audio-visual supervision, acoustic structure prediction, or reasoning from large multimodal models, requiring extra networks or strong inductive biases. Inspired by recent advances in visual representation learning, we introduce TD-V2A, which leverages temporal differences (TD) as the key representation that distinguishes V2A from I2A, enriching visual conditioning with minimal architectural modification. We first investigate TD at both the frame and feature levels to identify the most effective representation level at which TD complements visual representations. Based on these findings, we develop a hierarchically continual learning strategy and an annealed temporal differences guidance method to progressively learn and exploit TD information during diffusion training and sampling process, respectively. Extensive experiments on benchmark datasets demonstrate that effectively exploiting TD through our proposed framework significantly improves end-to-end V2A generation quality, even outperforming dedicated V2A representations such as contrastive audio-visual pretraining.
