InsertAnywhere：基于几何和光学的视频物体插入
InsertAnywhere: Geometrically Grounded and Optics-Aware Video Object Insertion
AI总结：
本文提出InsertAnywhere框架，通过4D感知遮罩生成模块实现几何准确的物体放置，并引入光学感知表示对齐策略生成逼真光学效果，结合开放的ROSE++数据集提升复杂场景下的视频物体插入效果。
摘要：
最近扩散模型的进步使视频编辑能力显著提升，但生产级视频物体插入（VOI）仍因缺乏4D场景理解和光学交互（如阴影和反射）而具有挑战性。为解决这些问题，我们提出了InsertAnywhere，一个全面的VOI框架，实现了基于几何的物体放置和光学感知的视频合成。我们的方法首先利用4D感知遮罩生成模块，允许用户在单帧中锚定物体的3D姿态。该框架自动将此放置传播到整个视频中，准确处理局部场景动态和遮挡。为了合成逼真的物理光照交互，我们引入了光学感知表示对齐策略，一种新的策略，利用扩展遮罩指导特征提取，使光学效果能够无缝延伸到插入物体边界之外。最后，为克服此类现象的训练数据缺乏，我们构建并开源了ROSE++，一个专门针对光学效果监督学习的四元组数据集。大量实验表明，InsertAnywhere在复杂现实场景中生成几何合理且光度逼真的插入效果，显著优于现有研究和商业生成工具。
英文摘要：
Recent advances in diffusion models have enabled impressive video editing capabilities, yet production-grade Video Object Insertion (VOI) remains challenging due to inadequate 4D scene understanding and a lack of proper optical interactions, such as shadows and reflections. To address these limitations, we present InsertAnywhere, a comprehensive VOI framework that achieves geometrically grounded object placement and optics-aware video synthesis. Our approach first leverages a 4D-aware mask generation module that allows users to anchor an object's 3D pose in a single frame. The framework automatically propagates this placement across the video, accurately handling local scene dynamics and occlusions. To synthesize realistic physical lighting interactions, we introduce Optics-Aware Representation Alignment, a novel strategy that utilizes an extended mask to guide feature extraction, enabling optical effects to seamlessly extend beyond the inserted object's boundary. Finally, to overcome the lack of training data for such phenomena, we construct and open-source ROSE++, a specialized quadruplet dataset tailored for the supervised learning of optical effects. Extensive experiments demonstrate that InsertAnywhere produces geometrically plausible and photometrically realistic insertions in complex real-world scenarios, significantly outperforming existing research and commercial generative tools.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
