ReCapture: 使用掩码视频微调为用户提供的视频生成视频摄像机控制
ReCapture: Generative Video Camera Controls for User-Provided Videos using Masked Video Fine-Tuning
AI总结：
本文提出ReCapture，一种从用户提供的单个视频生成新视角视频的方法，通过多视角扩散或点云渲染生成锚定视频，再经掩码视频微调得到干净一致的结果。
摘要：
最近，视频建模的突破使得在生成的视频中能够控制摄像机轨迹。然而，这些方法不能直接应用于非视频模型生成的用户提供的视频。在本文中，我们提出了ReCapture，一种从单个用户提供的视频生成具有新颖摄像机轨迹的新视频的方法。我们的方法允许我们从截然不同的角度和电影感的摄像机运动重新生成参考视频，并保留其所有现有的场景运动。值得注意的是，使用我们的方法，我们还可以合理地幻觉出参考视频中不可观察的场景部分。我们的方法通过以下步骤工作：(1) 使用多视角扩散模型或基于深度的点云渲染生成具有新摄像机轨迹的噪声锚定视频，然后(2) 使用我们提出的掩码视频微调技术将锚定视频重新生成为干净且时间上一致的重新角度视频。
英文摘要：
Recently, breakthroughs in video modeling have allowed for controllable camera trajectories in generated videos. However, these methods cannot be directly applied to user-provided videos that are not generated by a video model. In this paper, we present ReCapture, a method for generating new videos with novel camera trajectories from a single user-provided video. Our method allows us to re-generate the reference video, with all its existing scene motion, from vastly different angles and with cinematic camera motion. Notably, using our method we can also plausibly hallucinate parts of the scene that were not observable in the reference video. Our method works by (1) generating a noisy anchor video with a new camera trajectory using multiview diffusion models or depth-based point cloud rendering and then (2) regenerating the anchor video into a clean and temporally consistent reangled video using our proposed masked video fine-tuning technique.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
