Smart-Insertion-V: 通过闭环反馈双流框架实现逼真的视频插入
Smart-Insertion-V: Photorealistic Video Insertion via a Closed-Loop Feedback Dual-Stream Framework
AI总结：
提出一种端到端双流框架Smart-Insertion-V，通过闭环反馈机制和双世界视角旋转位置编码解决参考对象与源视频风格域差异大的问题，实现无需掩码的逼真视频对象插入。
摘要：
无掩码视频对象插入已成为一项具有挑战性的任务，需要将参考对象和谐地融入源视频中。然而，当参考对象与源场景存在严重的风格域差异时，现有方法难以应对。为了克服这一问题，我们提出了 extit{ extbf{Smart-Insertion-V}}，一种端到端的 extbf{双流}框架，同时进行视频插入和图像风格迁移。在该框架内，图像流同步引导视频生成过程，同时进一步引入 extbf{闭环反馈}机制以确保鲁棒插入。不可避免地，整合这些多样化的条件信号会导致特征纠缠和风格泄露。为解决此问题，我们设计了 extbf{双世界视角旋转位置编码}，通过时空偏移区分不同信号，且不增加大量训练开销。此外，为了促进空间定位和风格适应，我们引入了 extbf{解耦引导模块}，该模块利用视觉语言模型进行语义推理，同时通过原生文本编码器保留原始时间引导。为了弥合和谐参考插入任务的数据差距，我们提出了一种数据整理流程，并将发布一个 extbf{开源数据集}。实验表明，我们的方法可以将对象插入到合理的位置，同时实现最和谐的结果。
英文摘要：
Mask-free video object insertion has emerged as a challenging task, requiring harmonious integration of reference objects into source videos. However, existing methods struggle when references exhibit severe stylistic domain gaps with the source scene. To overcome this, we propose \textit{\textbf{Smart-Insertion-V}}, an end-to-end \textbf{Dual-Stream} framework that concurrently conducts video insertion and image style transfer. Within this framework, the image stream synchronously guides the video generation process, while a \textbf{Closed-loop Feedback} mechanism is further incorporated to ensure robust insertion. Inevitably, integrating these diverse conditioning signals results in feature entanglement and style leakage. To tackle this issue, we design \textbf{Dual-World-View RoPE} to distinguish different signals via spatial-temporal offsets without incurring heavy training overhead. Furthermore, to facilitate spatial grounding and stylistic adaptation, we introduce a \textbf{Decoupled Guidance Module} that leverages a Vision-Language Model for semantic reasoning while preserving original temporal guidance with native text encoder. To bridge data gap for harmonious reference insertion task, we propose a data curation pipeline and will release an \textbf{open-source dataset}. Experiments demonstrate that our method can insert objects into plausible positions while achieving the most harmonious results.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
