Rhythmic Foley：视频转音频合成中实现音视频无缝对齐的框架
Rhythmic Foley: A Framework For Seamless Audio-Visual Alignment In Video-to-Audio Synthesis
AI总结：
本研究提出Rhythmic Foley视频转音频合成框架，通过语义对齐与时间同步双适配器及对比音视频预训练编码器，解决音视频不同步和语义丢失问题，提升生成音频质量与可控性，实验验证了其音视频无缝对齐的有效性。
摘要：
本研究提出了一种创新的视频转音频合成框架，解决了音视频不同步及音频语义丢失的问题。通过引入语义对齐适配器和时间同步适配器，我们的方法显著提升了语义完整性和节拍点同步精度，尤其在快节奏动作序列中效果突出。该模型采用对比音视频预训练编码器，使用视频与高质量音频数据进行训练，提升了生成音频的质量。这种双适配器方法让用户能更好地控制音频语义和节拍效果，可通过调整控制器获得更优结果。大量实验证实了该框架在实现音视频无缝对齐方面的有效性。
英文摘要：
Our research introduces an innovative framework for video-to-audio synthesis, which solves the problems of audio-video desynchronization and semantic loss in the audio. By incorporating a semantic alignment adapter and a temporal synchronization adapter, our method significantly improves semantic integrity and the precision of beat point synchronization, particularly in fast-paced action sequences. Utilizing a contrastive audio-visual pre-trained encoder, our model is trained with video and high-quality audio data, improving the quality of the generated audio. This dual-adapter approach empowers users with enhanced control over audio semantics and beat effects, allowing the adjustment of the controller to achieve better results. Extensive experiments substantiate the effectiveness of our framework in achieving seamless audio-visual alignment.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
