arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2409.17550cs.LGcs.MMcs.SDeess.AS
收藏

一个简单但强大的有声视频生成基线:音频与视频扩散模型面向联合生成的有效适配

A Simple but Strong Baseline for Sounding Video Generation: Effective Adaptation of Audio and Video Diffusion Models for Joint Generation

Sony AI(索尼人工智能);Sony Group Corp.(索尼集团公司)

Masato Ishii, Akio Hayakawa, Takashi Shibuya, Yuki Mitsufuji

AI总结:

本文提出一个有声视频联合生成的简单强基线,通过时间步调整和CMC-PE两种机制增强音视频时间对齐,实验表明其优于现有方法。

摘要:

在这项工作中,我们构建了一个简单但强大的有声视频生成基线。给定音频和视频的基础扩散模型,我们将它们与额外模块集成到单一模型中,并训练该模型以联合生成音频和视频。为了增强音频-视频对之间的对齐,我们在模型中引入了两种新颖机制。第一种是时间步调整(timestep adjustment),它为每个基础模型提供不同的时间步信息,旨在使样本生成过程在跨模态的时间步上保持一致。第二种是额外模块的新设计,称为跨模态条件作为位置编码(Cross-Modal Conditioning as Positional Encoding,CMC-PE)。在CMC-PE中,跨模态信息被嵌入为仿佛表示时间位置信息,并且这些嵌入像位置编码一样被输入到模型中。与流行的交叉注意力机制相比,CMC-PE为生成数据中的时间对齐提供了更好的归纳偏置。实验结果验证了这两种新引入机制的有效性,并表明我们的方法优于现有方法。

英文摘要:

In this work, we build a simple but strong baseline for sounding video generation. Given base diffusion models for audio and video, we integrate them with additional modules into a single model and train it to make the model jointly generate audio and video. To enhance alignment between audio-video pairs, we introduce two novel mechanisms in our model. The first one is timestep adjustment, which provides different timestep information to each base model. It is designed to align how samples are generated along with timesteps across modalities. The second one is a new design of the additional modules, termed Cross-Modal Conditioning as Positional Encoding (CMC-PE). In CMC-PE, cross-modal information is embedded as if it represents temporal position information, and the embeddings are fed into the model like positional encoding. Compared with the popular cross-attention mechanism, CMC-PE provides a better inductive bias for temporal alignment in the generated data. Experimental results validate the effectiveness of the two newly introduced mechanisms and also demonstrate that our method outperforms existing methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑