Zero-to-Hero：基于零样本初始化的参考图引导视频外观编辑
Zero-to-Hero: Zero-Shot Initialization Empowering Reference-Based Video Appearance Editing
AI总结：
本文提出Zero-to-Hero方法，通过解耦编辑过程为锚点帧编辑与外观传播两阶段，利用原始帧对应关系引导注意力机制实现零样本初始化，并引入视频恢复模型解决成像退化问题，在PSNR上提升2.6 dB。
摘要：
根据用户需求进行外观编辑是视频编辑中的一项关键任务。现有的文本引导方法常导致用户意图模糊，并限制了对物体特定方面的细粒度控制。为克服这些局限，本文提出了一种名为Zero-to-Hero的新方法，专注于基于参考图的视频编辑，将编辑过程解耦为两个独立的问题。该方法首先编辑一个锚点帧以满足用户需求作为参考图像，然后将其外观一致地传播到其他帧。我们利用原始帧内的对应关系来引导注意力机制，这在内存友好的视频生成模型中比以往提出的基于光流或时间模块的方法更鲁棒，尤其是在处理大运动物体时。它提供了一个可靠的零样本初始化，确保了准确性和时间一致性。然而，对注意力机制的干预会导致颜色过饱和和未知模糊等复合成像退化问题。从Zero-Stage开始，我们的Hero-Stage整体学习一个用于视频恢复的条件生成模型。为准确评估外观一致性，我们使用Blender构建了一组具有多种外观的视频，实现了细粒度且确定性的评估。我们的方法优于表现最佳的基线方法，PSNR提升了2.6 dB。项目页面位于https://github.com/Tonniia/Zero2Hero。
英文摘要：
Appearance editing according to user needs is a pivotal task in video editing. Existing text-guided methods often lead to ambiguities regarding user intentions and restrict fine-grained control over editing specific aspects of objects. To overcome these limitations, this paper introduces a novel approach named {Zero-to-Hero}, which focuses on reference-based video editing that disentangles the editing process into two distinct problems. It achieves this by first editing an anchor frame to satisfy user requirements as a reference image and then consistently propagating its appearance across other frames. We leverage correspondence within the original frames to guide the attention mechanism, which is more robust than previously proposed optical flow or temporal modules in memory-friendly video generative models, especially when dealing with objects exhibiting large motions. It offers a solid ZERO-shot initialization that ensures both accuracy and temporal consistency. However, intervention in the attention mechanism results in compounded imaging degradation with over-saturated colors and unknown blurring issues. Starting from Zero-Stage, our Hero-Stage Holistically learns a conditional generative model for vidEo RestOration. To accurately evaluate the consistency of the appearance, we construct a set of videos with multiple appearances using Blender, enabling a fine-grained and deterministic evaluation. Our method outperforms the best-performing baseline with a PSNR improvement of 2.6 dB. The project page is at https://github.com/Tonniia/Zero2Hero.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
