EditWorld：模拟世界动态以实现指令遵循的图像编辑
EditWorld: Simulating World Dynamics for Instruction-Following Image Editing
AI总结：
针对现有图像编辑缺乏对物理世界动态理解的问题，提出EditWorld任务及数据集，通过大型预训练模型构建数据并训练模型，显著提升了世界指令图像编辑能力。
摘要：
扩散模型显著提升了图像编辑的性能。现有方法实现了多种途径以达成高质量图像编辑，包括但不限于文本控制、拖拽操作以及掩码与修复。其中，基于指令的编辑因其在多样场景下遵循人类指令的便捷性与有效性而脱颖而出。然而，它仍局限于添加、替换或删除等简单编辑操作，且在理解传达物理世界现实动态特性的世界动态方面存在不足。因此，本工作EditWorld引入了一项新编辑任务，即世界指令图像编辑，该任务定义并分类了基于各种世界场景的指令。我们使用一组大型预训练模型（例如，GPT-3.5、Video-LLava和SDXL）构建了一个带有世界指令的新图像编辑数据集。为了对图像编辑实现充分的世界动态模拟，我们的EditWorld在精心构建的数据集上训练模型，并通过设计的编辑后策略提升指令遵循能力。大量实验表明，我们的方法在这一新任务上显著优于现有编辑方法。
英文摘要：
Diffusion models have significantly improved the performance of image editing. Existing methods realize various approaches to achieve high-quality image editing, including but not limited to text control, dragging operation, and mask-and-inpainting. Among these, instruction-based editing stands out for its convenience and effectiveness in following human instructions across diverse scenarios. However, it still focuses on simple editing operations like adding, replacing, or deleting, and falls short of understanding aspects of world dynamics that convey the realistic dynamic nature in the physical world. Therefore, this work, EditWorld, introduces a new editing task, namely world-instructed image editing, which defines and categorizes the instructions grounded by various world scenarios. We curate a new image editing dataset with world instructions using a set of large pretrained models (e.g., GPT-3.5, Video-LLava and SDXL). To enable sufficient simulation of world dynamics for image editing, our EditWorld trains model in the curated dataset, and improves instruction-following ability with designed post-edit strategy. Extensive experiments demonstrate our method significantly outperforms existing editing methods in this new task. Our dataset and code will be available at https://github.com/YangLing0818/EditWorld
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
