InstructX：迈向基于MLLM引导的统一视觉编辑
InstructX: Towards Unified Visual Editing with MLLM Guidance
AI总结：
本文提出统一框架InstructX，通过全面研究MLLM与扩散模型集成，利用图像训练涌现视频编辑能力并融合模态特定特征，在单一模型中实现图像和视频编辑，达到最先进性能。
摘要：
随着多模态大语言模型（MLLMs）在视觉理解和推理方面展现出强大能力，利用它们来提升扩散模型编辑性能的研究兴趣日益增长。尽管进展迅速，大多数研究缺乏对MLLM设计选择的深入分析。此外，在一些困难任务（如视频编辑）中，MLLM与扩散模型的集成仍是一个开放挑战。在本文中，我们提出了InstructX，一个用于图像和视频编辑的统一框架。具体而言，我们对跨多种任务的指令驱动编辑中集成MLLM和扩散模型进行了全面研究。基于此研究，我们分析了图像和视频在统一建模中的协同与区别。（1）我们表明，在图像数据上训练可以产生无需显式监督的涌现式视频编辑能力，从而缓解了视频训练数据稀缺带来的限制。（2）通过结合模态特定的MLLM特征，我们的方法有效地在单一模型中统一了图像和视频编辑任务。大量实验表明，我们的方法能够处理广泛的图像和视频编辑任务，并取得了最先进的性能。
英文摘要：
With recent advances in Multimodal Large Language Models (MLLMs) showing strong visual understanding and reasoning, interest is growing in using them to improve the editing performance of diffusion models. Despite rapid progress, most studies lack an in-depth analysis of MLLM design choices. Moreover, the integration of MLLMs and diffusion models remains an open challenge in some difficult tasks, such as video editing. In this paper, we present InstructX, a unified framework for image and video editing. Specifically, we conduct a comprehensive study on integrating MLLMs and diffusion models for instruction-driven editing across diverse tasks. Building on this study, we analyze the cooperation and distinction between images and videos in unified modeling. (1) We show that training on image data can lead to emergent video editing capabilities without explicit supervision, thereby alleviating the constraints imposed by scarce video training data. (2) By incorporating modality-specific MLLM features, our approach effectively unifies image and video editing tasks within a single model. Extensive experiments demonstrate that our method can handle a broad range of image and video editing tasks and achieves state-of-the-art performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
