OmniDrag：实现全向图像到视频生成的运动控制
OmniDrag: Enabling Motion Control for Omnidirectional Image-to-Video Generation
AI总结：
针对全向视频生成依赖文本导致内容不准、现有运动控制方法直接应用易失真的问题，提出OmniDrag框架，通过全向控制模块、球面运动估计器及Move360数据集，实现场景与对象级的高质量全向图像到视频运动控制。
摘要：
随着虚拟现实日益普及，对沉浸式动态全向视频（ODV）的可控创作需求正不断增长。尽管以往的文本到ODV生成方法取得了令人瞩目的成果，但由于仅依赖文本输入，它们存在内容不准确和不一致的问题。虽然近期的运动控制技术为视频生成提供了细粒度控制，但将这些方法直接应用于ODV往往会导致空间失真和效果不佳，尤其是在复杂球面运动的情况下。为解决这些挑战，我们提出了OmniDrag，这是首个同时支持场景级和对象级运动控制、以生成准确高质量全向图像到视频的方法。基于预训练视频扩散模型，我们引入了一个全向控制模块，该模块与时间注意力层联合微调，以有效处理复杂球面运动。此外，我们开发了一种新颖的球面运动估计器，可准确提取运动控制信号，让用户只需绘制手柄点和目标点即可实现拖拽式ODV生成。我们还推出了一个名为Move360的新数据集，以解决具有大幅场景和对象运动的ODV数据稀缺问题。实验表明，OmniDrag在实现ODV生成的整体场景级和细粒度对象级控制方面具有显著优势。项目页面位于https://lwq20020127.github.io/OmniDrag。
英文摘要：
As virtual reality gains popularity, the demand for controllable creation of immersive and dynamic omnidirectional videos (ODVs) is increasing. While previous text-to-ODV generation methods achieve impressive results, they struggle with content inaccuracies and inconsistencies due to reliance solely on textual inputs. Although recent motion control techniques provide fine-grained control for video generation, directly applying these methods to ODVs often results in spatial distortion and unsatisfactory performance, especially with complex spherical motions. To tackle these challenges, we propose OmniDrag, the first approach enabling both scene- and object-level motion control for accurate, high-quality omnidirectional image-to-video generation. Building on pretrained video diffusion models, we introduce an omnidirectional control module, which is jointly fine-tuned with temporal attention layers to effectively handle complex spherical motion. In addition, we develop a novel spherical motion estimator that accurately extracts motion-control signals and allows users to perform drag-style ODV generation by simply drawing handle and target points. We also present a new dataset, named Move360, addressing the scarcity of ODV data with large scene and object motions. Experiments demonstrate the significant superiority of OmniDrag in achieving holistic scene-level and fine-grained object-level control for ODV generation. The project page is available at https://lwq20020127.github.io/OmniDrag.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
