arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2604.06010cs.CV
收藏

OmniCamera:一个统一的多任务视频生成框架,支持任意相机控制

OmniCamera: A Unified Framework for Multi-task Video Generation with Arbitrary Camera Control

Sun Yat-sen University(中山大学);Hunyuan, Tencent(腾讯混元);CityU(香港城市大学);PolyU(香港理工大学)

Yukun Wang, Ruihuang Li, Jiale Tao, Shiyuan Yang, Liyi Chen, Zhantao Yang, Handz, Yulan Guo, Shuai Shao, Qinglin Lu

AI总结:

OmniCamera通过显式解耦场景内容与相机运动,实现灵活的视频生成,提出混合数据集和双阶段课程协同训练策略,提升多任务学习性能。

摘要:

视频本质上融合了两个关键维度:场景的动态内容和通过其观察的相机运动。然而,现有生成模型往往将这些因素纠缠在一起,限制了独立控制。本文引入OmniCamera统一框架,旨在显式解耦并控制这两个维度。这种组合方法通过允许任意的相机和内容条件配对,实现灵活的视频生成,解锁前所未有的创作控制。为克服此类系统固有的模态冲突和数据稀缺问题,我们提出了两个关键创新。首先,我们构建了OmniCAM,一个新颖的混合数据集,结合精心挑选的现实视频和合成数据,提供多样化的配对示例以实现稳健的多任务学习。其次,我们提出了一种双阶段课程协同训练策略,该策略通过逐步引入难度递增的控制模态(条件级)和在合成数据上训练精确控制后再适应真实数据以实现逼真效果(数据级),从而缓解模态干扰并协同学习于多样化的数据源。结果表明,OmniCamera实现了最先进的性能,能够在复杂相机运动中实现灵活控制,同时保持卓越的视觉质量。

英文摘要:

Video fundamentally intertwines two crucial axes: the dynamic content of a scene and the camera motion through which it is observed. However, existing generation models often entangle these factors, limiting independent control. In this work, we introduce OmniCamera, a unified framework designed to explicitly disentangle and command these two dimensions. This compositional approach enables flexible video generation by allowing arbitrary pairings of camera and content conditions, unlocking unprecedented creative control. To overcome the fundamental challenges of modality conflict and data scarcity inherent in such a system, we present two key innovations. First, we construct OmniCAM, a novel hybrid dataset combining curated real-world videos with synthetic data that provides diverse paired examples for robust multi-task learning. Second, we propose a Dual-level Curriculum Co-Training strategy that mitigates modality interference and synergistically learns from diverse data sources. This strategy operates on two levels: first, it progressively introduces control modalities by difficulties (condition-level), and second, trains for precise control on synthetic data before adapting to real data for photorealism (data-level). As a result, OmniCamera achieves state-of-the-art performance, enabling flexible control for complex camera movements while maintaining superior visual quality.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑