arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2501.02487cs.CV
收藏

ACE++: 基于指令的图像生成与编辑通过上下文感知内容填充实现

ACE++: Instruction-Based Image Creation and Editing via Context-Aware Content Filling

Tongyi Lab(通义实验室)

Chaojie Mao, Jingfeng Zhang, Yulin Pan, Zeyinzi Jiang, Zhen Han, Yu Liu, Jingren Zhou

AI总结:

本研究提出ACE++指令式扩散框架,通过改进长上下文条件单元、采用两阶段训练方案,实现多场景图像生成与编辑,在图像质量和提示遵循性上表现更优。

摘要:

我们提出了ACE++,这是一个基于指令的扩散框架,可处理各类图像生成与编辑任务。受FLUX.1-Fill-dev提出的图像修复任务输入格式启发,我们改进了ACE中引入的长上下文条件单元(LCU),并将该输入范式扩展至所有编辑与生成任务。为充分利用图像生成先验,我们开发了两阶段训练方案,以最小化对FLUX.1-dev这类强大文生图扩散模型进行微调的工作量。第一阶段,我们使用任务数据结合文生图模型的0-ref任务对模型进行预训练。社区中有许多基于文生图基础模型后训练的模型符合这一第一阶段训练范式,例如主要处理修复任务的FLUX.1-Fill-dev,可作为初始化模型以加速训练过程。第二阶段,我们使用ACE中定义的所有任务对上述模型进行微调,使其支持通用指令。为推动ACE++在不同场景中的广泛应用,我们提供了涵盖全量微调和轻量级微调的完整模型集,同时兼顾通用适用性和垂直场景适用性。定性分析表明,ACE++在生成图像质量和提示遵循能力方面具有优势。代码和模型将在项目页面发布:https://ali-vilab.github.io/ACE_plus_page/。

英文摘要:

We report ACE++, an instruction-based diffusion framework that tackles various image generation and editing tasks. Inspired by the input format for the inpainting task proposed by FLUX.1-Fill-dev, we improve the Long-context Condition Unit (LCU) introduced in ACE and extend this input paradigm to any editing and generation tasks. To take full advantage of image generative priors, we develop a two-stage training scheme to minimize the efforts of finetuning powerful text-to-image diffusion models like FLUX.1-dev. In the first stage, we pre-train the model using task data with the 0-ref tasks from the text-to-image model. There are many models in the community based on the post-training of text-to-image foundational models that meet this training paradigm of the first stage. For example, FLUX.1-Fill-dev deals primarily with painting tasks and can be used as an initialization to accelerate the training process. In the second stage, we finetune the above model to support the general instructions using all tasks defined in ACE. To promote the widespread application of ACE++ in different scenarios, we provide a comprehensive set of models that cover both full finetuning and lightweight finetuning, while considering general applicability and applicability in vertical scenarios. The qualitative analysis showcases the superiority of ACE++ in terms of generating image quality and prompt following ability. Code and models will be available on the project page: https://ali-vilab. github.io/ACE_plus_page/.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑