时间回声:解锁视频到音频生成模型中的长度泛化
Echoes Over Time: Unlocking Length Generalization in Video-to-Audio Generation Models
AI总结:
本文提出MMHNet模型,通过层级结构和非因果Mamba实现长音频生成,提升生成长度至5分钟以上,证明短训练长测试的可行性,优于现有视频到音频生成方法。
摘要:
多模态对齐的扩展在视频到音频生成中具有挑战性,尤其受限于数据量和文本描述与帧级视频信息的不匹配。本文研究了在短实例上训练的模型能否在测试时泛化到更长的实例。我们提出多模态层级网络MMHNet,作为最先进的视频到音频模型的增强扩展。我们的方法整合了层级方法和非因果Mamba以支持长音频生成。我们的方法显著提升了长音频生成,达到超过5分钟。我们证明在视频到音频生成任务中,无需训练更长的持续时间,短训练长测试是可能的。我们在实验中展示了我们的方法在长视频到音频基准上的显著成果,优于现有视频到音频任务的方法。此外,我们展示了模型在生成超过5分钟音频方面的能力,而先前的视频到音频方法在生成长持续时间时表现不足。
英文摘要:
Scaling multimodal alignment between video and audio is challenging, particularly due to limited data and the mismatch between text descriptions and frame-level video information. In this work, we tackle the scaling challenge in multimodal-to-audio generation, examining whether models trained on short instances can generalize to longer ones during testing. To tackle this challenge, we present multimodal hierarchical networks so-called MMHNet, an enhanced extension of state-of-the-art video-to-audio models. Our approach integrates a hierarchical method and non-causal Mamba to support long-form audio generation. Our proposed method significantly improves long audio generation up to more than 5 minutes. We also prove that training short and testing long is possible in the video-to-audio generation tasks without training on the longer durations. We show in our experiments that our proposed method could achieve remarkable results on long-video to audio benchmarks, beating prior works in video-to-audio tasks. Moreover, we showcase our model capability in generating more than 5 minutes, while prior video-to-audio methods fall short in generating with long durations.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。