LoRA-Edit: 通过掩码感知LoRA微调实现可控的第一帧引导视频编辑
LoRA-Edit: Controllable First-Frame-Guided Video Editing via Mask-Aware LoRA Fine-Tuning
AI总结：
LoRA-Edit通过时空掩码引导LoRA微调，实现对视频编辑全过程的可控编辑，支持生成动态变化的视频内容。
摘要：
使用扩散模型的视频编辑在生成高质量视频编辑方面取得了显著成果。然而，当前方法往往依赖大规模预训练，限制了特定编辑的灵活性。第一帧引导编辑提供了对第一帧的控制，但缺乏对编辑后续时间演变的精细控制。为了解决这个问题，我们提出了一种基于掩码的LoRA（低秩适应）调优方法，用于适应预训练的图像到视频模型，以实现灵活的视频编辑。我们的关键创新是使用时空掩码来战略性地引导LoRA微调过程。这教会模型两种不同的能力：首先，将掩码解释为命令，要么保留源视频中的内容，要么在指定区域生成新内容。其次，对于这些生成区域，LoRA学会合成要么继承自视频的时间一致运动，要么由用户提供的参考帧引导的新外观。这种双能力LoRA使用户能够控制编辑的整个时间演变，允许复杂的变换，如物体旋转或花朵绽放。实验结果表明，我们的方法在视频编辑性能上优于基线方法。代码和视频结果可在我们的项目网站上获得：https://cjeen.github.io/LoRAEdit。
英文摘要：
Video editing using diffusion models has achieved remarkable results in generating high-quality edits for videos. However, current methods often rely on large-scale pretraining, limiting flexibility for specific edits. First-frame-guided editing provides control over the first frame, but lacks fine-grained control over the edit's subsequent temporal evolution. To address this, we propose a mask-based LoRA (Low-Rank Adaptation) tuning method that adapts pretrained Image-to-Video models for flexible video editing. Our key innovation is using a spatiotemporal mask to strategically guide the LoRA fine-tuning process. This teaches the model two distinct skills: first, to interpret the mask as a command to either preserve content from the source video or generate new content in designated regions. Second, for these generated regions, LoRA learns to synthesize either temporally consistent motion inherited from the video or novel appearances guided by user-provided reference frames. This dual-capability LoRA grants users control over the edit's entire temporal evolution, allowing complex transformations like an object rotating or a flower blooming. Experimental results show our method achieves superior video editing performance compared to baseline methods. The code and video results are available at our project website: https://cjeen.github.io/LoRAEdit.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
