POEM：通过MLLM控制实现精确的对象级编辑
POEM: Precise Object-level Editing via MLLM control
AI总结：
本文提出POEM框架，利用多模态大语言模型生成变换前后的精确对象掩码来指导扩散模型进行对象级编辑，并构建VOCEdits基准，在减少人工操作的同时提升编辑精度与可靠性。
摘要：
扩散模型显著提升了文本到图像生成能力，能够根据文本描述生成高质量、逼真的图像。除生成之外，对象级图像编辑仍然是一个具有挑战性的问题，它要求在保持视觉一致性的同时进行精确修改。现有的基于文本指令的编辑方法难以处理局部形状和布局变换，常常引入非预期的全局变化。基于图像交互的方法虽然精度更高，但需要人工提供精确指导。为了在保持较高图像编辑精度的同时减少人工操作，本文提出了POEM，一个利用多模态大语言模型（MLLMs）进行精确对象级编辑的框架。POEM利用MLLMs分析指令提示，并在变换前后生成精确的对象掩码，从而在无需大量用户输入的情况下实现细粒度控制。这一结构化推理阶段指导基于扩散的编辑过程，确保准确的对象定位与变换。为评估所提方法，我们引入了VOCEdits，一个基于PASCAL VOC 2012构建的基准数据集，并补充了指令性编辑提示、真值变换和精确对象掩码。实验结果表明，POEM在精度和可靠性上优于现有的基于文本的图像编辑方法，同时与基于交互的方法相比减少了人工操作。
英文摘要：
Diffusion models have significantly improved text-to-image generation, producing high-quality, realistic images from textual descriptions. Beyond generation, object-level image editing remains a challenging problem, requiring precise modifications while preserving visual coherence. Existing text-based instructional editing methods struggle with localized shape and layout transformations, often introducing unintended global changes. Image interaction-based approaches offer better accuracy but require manual human effort to provide precise guidance. To reduce this manual effort while maintaining a high image editing accuracy, in this paper, we propose POEM, a framework for Precise Object-level Editing using Multimodal Large Language Models (MLLMs). POEM leverages MLLMs to analyze instructional prompts and generate precise object masks before and after transformation, enabling fine-grained control without extensive user input. This structured reasoning stage guides the diffusion-based editing process, ensuring accurate object localization and transformation. To evaluate our approach, we introduce VOCEdits, a benchmark dataset based on PASCAL VOC 2012, augmented with instructional edit prompts, ground-truth transformations, and precise object masks. Experimental results show that POEM outperforms existing text-based image editing approaches in precision and reliability while reducing manual effort compared to interaction-based methods.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
