Show Me: 用扩散模型统一指令图像和视频生成
Show Me: Unifying Instructional Image and Video Generation with Diffusion Models
AI总结：
ShowMe通过统一视频扩散模型的空间和时间组件，提升指令图像和视频生成的性能与一致性。
摘要：
在给定的上下文中生成视觉指令对于开发交互式世界模拟器至关重要。尽管先前的工作通过文本引导的图像编辑或视频预测来解决这个问题，但这些任务通常是孤立处理的。这种分离暴露了一个根本性的问题：图像编辑方法忽略了动作随时间展开的方式，而视频预测模型往往忽略了预期的结果。为此，我们提出了ShowMe，一个统一的框架，通过选择性地激活视频扩散模型的空间和时间组件，同时引入结构和运动一致性奖励以提高结构保真度和时间一致性。值得注意的是，这种统一带来了双重好处：通过视频预训练获得的空间知识增强了非刚性图像编辑中的上下文一致性和真实性，而指令引导的编辑阶段使模型具备更强的目标导向推理能力用于视频预测。在多样化的基准测试中，我们的方法在指令图像和视频生成方面均优于专家模型，突显了视频扩散模型作为统一的动作-对象状态转换器的强大力量。
英文摘要：
Generating visual instructions in a given context is essential for developing interactive world simulators. While prior works address this problem through either text-guided image manipulation or video prediction, these tasks are typically treated in isolation. This separation reveals a fundamental issue: image manipulation methods overlook how actions unfold over time, while video prediction models often ignore the intended outcomes. To this end, we propose ShowMe, a unified framework that enables both tasks by selectively activating the spatial and temporal components of video diffusion models. In addition, we introduce structure and motion consistency rewards to improve structural fidelity and temporal coherence. Notably, this unification brings dual benefits: the spatial knowledge gained through video pretraining enhances contextual consistency and realism in non-rigid image edits, while the instruction-guided manipulation stage equips the model with stronger goal-oriented reasoning for video prediction. Experiments on diverse benchmarks demonstrate that our method outperforms expert models in both instructional image and video generation, highlighting the strength of video diffusion models as a unified action-object state transformer.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
