Lego-Edit：基于模型级积木与MLLM构建器的通用图像编辑框架
Lego-Edit: A General Image Editing Framework with Model-Level Bricks and MLLM Builder
AI总结：
提出Lego-Edit通用图像编辑框架，利用MLLM组织模型级编辑工具，通过三阶段渐进式强化学习实现开放域指令的泛化推理，在多个基准上达到SOTA。
摘要：
基于指令的图像编辑因其与用户的直接交互而备受关注。然而，现实世界的用户指令极其多样化，现有方法往往无法有效泛化到训练域之外的指令，限制了其实际应用。为此，我们提出Lego-Edit，利用多模态大语言模型（MLLM）的泛化能力来组织一套模型级编辑工具以应对该挑战。Lego-Edit包含两个关键设计：（1）模型级工具包，由在有限数据上高效训练的多种模型及若干图像操作函数组成，使MLLM能够实现编辑动作的细粒度组合；（2）三阶段渐进式强化学习方法，利用对未标注开放域指令的反馈来训练MLLM，赋予其处理现实世界指令的泛化推理能力。实验表明，Lego-Edit在GEdit-Bench和ImgBench上取得了最先进的性能。它对开放域指令展现出稳健的推理能力，并能在无需额外微调的情况下使用新引入的编辑工具。代码已开源：https://github.com/xiaomi-research/lego-edit。
英文摘要：
Instruction-based image editing has garnered significant attention due to its direct interaction with users. However, real-world user instructions are immensely diverse, and existing methods often fail to generalize effectively to instructions outside their training domain, limiting their practical application. To address this, we propose Lego-Edit, which leverages the generalization capability of Multi-modal Large Language Model (MLLM) to organize a suite of model-level editing tools to tackle this challenge. Lego-Edit incorporates two key designs: (1) a model-level toolkit comprising diverse models efficiently trained on limited data and several image manipulation functions, enabling fine-grained composition of editing actions by the MLLM; and (2) a three-stage progressive reinforcement learning approach that uses feedback on unannotated, open-domain instructions to train the MLLM, equipping it with generalized reasoning capabilities for handling real-world instructions. Experiments demonstrate that Lego-Edit achieves state-of-the-art performance on GEdit-Bench and ImgBench. It exhibits robust reasoning capabilities for open-domain instructions and can utilize newly introduced editing tools without additional fine-tuning. Code is available: https://github.com/xiaomi-research/lego-edit.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
