让您的视频聆听您的音乐！
Let Your Video Listen to Your Music!
AI总结：
本文提出MVAA框架，通过关键帧对齐与节奏感知视频修复两步法，自动将视频运动与音乐节拍对齐，保留原内容，10分钟内完成适应，实现高质量节拍对齐与视觉平滑。
摘要：
将视频中的视觉运动节奏与给定音乐曲目对齐是多媒体制作中的实际需求，但在自主视频编辑中仍是一项探索不足的任务。运动与音乐节拍之间的有效对齐能增强观众的参与度和视觉吸引力，尤其在音乐视频、宣传内容和电影剪辑中。现有方法通常依赖于劳动密集型的手动剪切、速度调整或基于启发式的编辑技术来实现同步。虽然一些生成模型能够处理视频和音乐的联合生成，但它们往往将两种模态纠缠在一起，限制了在保留完整视觉内容的同时将视频与音乐节拍对齐的灵活性。在本文中，我们提出了一种新颖且高效的框架，称为MVAA（音乐-视频自动对齐），它能够自动编辑视频以与给定音乐曲目的节奏对齐，同时保留原始视觉内容。为了增强灵活性，我们将任务模块化为MVAA中的两步过程：将运动关键帧与音频节拍对齐，随后进行节奏感知的视频修复。具体来说，我们首先在与音乐节拍对齐的时间戳处插入关键帧，然后使用帧条件扩散模型生成连贯的中间帧，保留原始视频的语义内容。由于全面的测试时训练可能耗时，我们采用两阶段策略：在小视频集上预训练修复模块以学习通用运动先验，然后进行快速的推理时微调以实现视频特定适应。这种混合方法能够在单个NVIDIA 4090 GPU上，以CogVideoX-5b-I2V为骨干，在一个epoch内于10分钟内完成适应。大量实验表明，我们的方法能够实现高质量的节拍对齐和视觉平滑度。
英文摘要：
Aligning the rhythm of visual motion in a video with a given music track is a practical need in multimedia production, yet remains an underexplored task in autonomous video editing. Effective alignment between motion and musical beats enhances viewer engagement and visual appeal, particularly in music videos, promotional content, and cinematic editing. Existing methods typically depend on labor-intensive manual cutting, speed adjustments, or heuristic-based editing techniques to achieve synchronization. While some generative models handle joint video and music generation, they often entangle the two modalities, limiting flexibility in aligning video to music beats while preserving the full visual content. In this paper, we propose a novel and efficient framework, termed MVAA (Music-Video Auto-Alignment), that automatically edits video to align with the rhythm of a given music track while preserving the original visual content. To enhance flexibility, we modularize the task into a two-step process in our MVAA: aligning motion keyframes with audio beats, followed by rhythm-aware video inpainting. Specifically, we first insert keyframes at timestamps aligned with musical beats, then use a frame-conditioned diffusion model to generate coherent intermediate frames, preserving the original video's semantic content. Since comprehensive test-time training can be time-consuming, we adopt a two-stage strategy: pretraining the inpainting module on a small video set to learn general motion priors, followed by rapid inference-time fine-tuning for video-specific adaptation. This hybrid approach enables adaptation within 10 minutes with one epoch on a single NVIDIA 4090 GPU using CogVideoX-5b-I2V as the backbone. Extensive experiments show that our approach can achieve high-quality beat alignment and visual smoothness.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
