Draw-In-Mind: 在统一多模态模型中重新平衡设计师-画家角色有助于图像编辑
Draw-In-Mind: Rebalancing Designer-Painter Roles in Unified Multimodal Models Benefits Image Editing
AI总结：
本文提出Draw-In-Mind模型，通过重新分配设计职责提升图像编辑性能，展示了在统一多模态模型中平衡理解与生成模块的重要性。
摘要：
近年来，将多模态理解和生成整合到单一统一模型中已成为一种有前途的范式。尽管这种范式在文本到图像（T2I）生成中取得了强劲成果，但仍难以实现精确的图像编辑。我们将其限制归因于责任分配的不平衡。理解模块主要作为翻译器，将用户指令编码为语义条件，而生成模块必须同时扮演设计师和画家，推断原始布局，识别目标编辑区域并渲染新内容。这种不平衡是反直觉的，因为理解模块通常用几倍的数据进行复杂推理任务的训练。为了解决这一问题，我们引入了Draw-In-Mind（DIM），一个包含两个互补子集的数据集：（i）DIM-T2I，包含1400万对长上下文图像-文本对以增强复杂指令理解；（ii）DIM-Edit，由GPT-4o生成的233,000个思考链想象组成，作为图像编辑的显式设计蓝图。我们通过轻量级的两层MLP将冻结的Qwen2.5-VL-3B与可训练的SANA1.5-1.6B连接起来，并在提出的DIM数据集上训练，得到DIM-4.6B-T2I/Edit。尽管参数规模较小，DIM-4.6B-Edit在ImgEdit和GEdit-Bench基准测试中实现了SOTA或具有竞争力的性能，优于UniWorld-V1和Step1X-Edit等更大模型。这些发现表明，显式地将设计责任分配给理解模块对图像编辑有显著好处。我们的数据集和模型可在https://github.com/showlab/DIM上获得。
英文摘要：
In recent years, integrating multimodal understanding and generation into a single unified model has emerged as a promising paradigm. While this approach achieves strong results in text-to-image (T2I) generation, it still struggles with precise image editing. We attribute this limitation to an imbalanced division of responsibilities. The understanding module primarily functions as a translator that encodes user instructions into semantic conditions, while the generation module must simultaneously act as designer and painter, inferring the original layout, identifying the target editing region, and rendering the new content. This imbalance is counterintuitive because the understanding module is typically trained with several times more data on complex reasoning tasks than the generation module. To address this issue, we introduce Draw-In-Mind (DIM), a dataset comprising two complementary subsets: (i) DIM-T2I, containing 14M long-context image-text pairs to enhance complex instruction comprehension; and (ii) DIM-Edit, consisting of 233K chain-of-thought imaginations generated by GPT-4o, serving as explicit design blueprints for image edits. We connect a frozen Qwen2.5-VL-3B with a trainable SANA1.5-1.6B via a lightweight two-layer MLP, and train it on the proposed DIM dataset, resulting in DIM-4.6B-T2I/Edit. Despite its modest parameter scale, DIM-4.6B-Edit achieves SOTA or competitive performance on the ImgEdit and GEdit-Bench benchmarks, outperforming much larger models such as UniWorld-V1 and Step1X-Edit. These findings demonstrate that explicitly assigning the design responsibility to the understanding module provides significant benefits for image editing. Our dataset and models are available at https://github.com/showlab/DIM.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
