在这项工作中,我们构建了一个简单但强大的有声视频生成基线。给定音频和视频的基础扩散模型,我们将它们与额外模块集成到单一模型中,并训练该模型以联合生成音频和视频。为了增强音频-视频对之间的对齐,我们在模型中引入了两种新颖机制。第一种是时间步调整(timestep adjustment),它为每个基础模型提供不同的时间步信息,旨在使样本生成过程在跨模态的时间步上保持一致。第二种是额外模块的新设计,称为跨模态条件作为位置编码(Cross-Modal Conditioning as Positional Encoding,CMC-PE)。在CMC-PE中,跨模态信息被嵌入为仿佛表示时间位置信息,并且这些嵌入像位置编码一样被输入到模型中。与流行的交叉注意力机制相比,CMC-PE为生成数据中的时间对齐提供了更好的归纳偏置。实验结果验证了这两种新引入机制的有效性,并表明我们的方法优于现有方法。
英文摘要:
In this work, we build a simple but strong baseline for sounding video generation. Given base diffusion models for audio and video, we integrate them with additional modules into a single model and train it to make the model jointly generate audio and video. To enhance alignment between audio-video pairs, we introduce two novel mechanisms in our model. The first one is timestep adjustment, which provides different timestep information to each base model. It is designed to align how samples are generated along with timesteps across modalities. The second one is a new design of the additional modules, termed Cross-Modal Conditioning as Positional Encoding (CMC-PE). In CMC-PE, cross-modal information is embedded as if it represents temporal position information, and the embeddings are fed into the model like positional encoding. Compared with the popular cross-attention mechanism, CMC-PE provides a better inductive bias for temporal alignment in the generated data. Experimental results validate the effectiveness of the two newly introduced mechanisms and also demonstrate that our method outperforms existing methods.
Peking University(北京大学);University of Surrey(萨里大学);UC Berkeley(加州大学伯克利分校);Tsinghua University(清华大学);CMU(卡内基梅隆大学);China Agricultural University(中国农业大学);MBZUAI(穆罕默德·本·扎耶德人工智能大学);Wuhan University of Technology(武汉理工大学)
Exploring Audio Cues for Enhanced Test-Time Video Model Adaptation
Artificial Intelligence Research Institute, Shenzhen MSU-BIT University and Guangdong-Hong Kong-Macao Joint Laboratory for Emotional Intelligence and Pervasive Computing(人工智能研究院、深圳MSU-BIT大学及粤港澳大湾区情感智能与普适计算联合实验室);School of Software Engineering, South China University of Technology(华南理工大学软件学院);College of Computing and Data Science, Nanyang Technological University(南洋理工大学计算与数据科学学院)
An Unsupervised Domain Adaptation Method for Locating Manipulated Region in partially fake Audio
University of Chinese Academy of Sciences(中国科学院大学);Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);Department of Automation, Tsinghua University(清华大学自动化系)