发表机构
Harbin Institute of Technology; Huawei（哈尔滨工业大学; 华为）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
针对视频世界模型天气控制不精确的问题，提出统一可控天气视频世界模型MeteoVerse，显式估计天气状态并利用转换感知专家实现保留、引入或移除天气效果，实验证明天气可控性显著提升。
AI 中文摘要
视频世界模型旨在根据观察到的场景，在遵循指定相机运动的同时预测未来内容。真实世界的场景演变不仅由视点变化和物体动态决定，还受到天气等环境条件的影响，这些条件会显著改变场景外观和可见性。建模这种逼真的天气演变具有挑战性，因为所需的天气修改取决于观察到的和期望的天气状态。根据它们之间的关系，模型可能需要保留、引入或移除某种天气效果。现有的视频世界模型通常将这种天气转换隐式化，迫使生成骨干网络将天气演变与场景动态和相机运动一起推断，导致天气控制不精确。为解决这一局限，我们提出了MeteoVerse，一个统一的可控天气视频世界模型，它从单张晴朗或恶劣天气图像生成未来视频，以无天气场景描述、目标天气指令和相机轨迹为条件。MeteoVerse不仅以期望天气为条件，还显式估计观察到的和目标天气状态，并表征所需的天气转换。一个转换感知的天气专家混合体随后将此转换转化为类别特定的残差天气特征，统一了天气保留、引入和移除，同时实现了对引入天气强度的细粒度控制。我们进一步构建了MeteoVerse数据集，包含超过5万个真实世界天气视频片段、生成的晴朗对应物、解耦的场景和天气描述、天气强度标注以及相机轨迹。大量实验表明，在保持有竞争力的场景一致性和相机控制性能的同时，天气可控性显著提高。
英文摘要
Video world models aim to predict future content from an observed scene while following prescribed camera motion. Real-world scene evolution is determined not only by changes in viewpoint and object dynamics, but also by environmental conditions such as weather, which can substantially alter scene appearance and visibility. Modeling such realistic weather evolution is challenging because the required weather modification depends jointly on the observed and desired weather states. Depending on their relation, the model may need to preserve, introduce, or remove a weather effect. Existing video world models typically leave this weather transition implicit, forcing the generation backbone to infer weather evolution together with scene dynamics and camera motion, which leads to imprecise weather control. To address this limitation, we propose MeteoVerse, a unified weather-controllable video world model that generates future videos from a single sunny or adverse-weather image, conditioned on a weather-free scene description, a target-weather instruction, and a camera trajectory. Rather than conditioning only on the desired weather, MeteoVerse explicitly estimates the observed and target weather states and represents the required weather transition. A transition-aware mixture of weather experts then translates this transition into category-specific residual weather features, unifying weather preservation, introduction, and removal while enabling fine-grained control over introduced weather intensity. We further construct the MeteoVerse dataset with over 50K real-world weather video clips, generated sunny counterparts, disentangled scene and weather descriptions, weather-intensity annotations, and camera trajectories. Extensive experiments demonstrate substantially improved weather controllability while retaining competitive scene consistency and camera-control performance.
Comments13 pages
