ACE：通过扩散Transformer实现的全能创作者与编辑器
ACE: All-round Creator and Editor Following Instructions via Diffusion Transformer
AI总结：
提出ACE，一种基于扩散Transformer的统一视觉生成模型，通过长上下文条件单元和高效数据收集，在多种生成与编辑任务上达到专家级性能，并支持多模态聊天系统。
摘要：
扩散模型已成为一种强大的生成技术，并被发现适用于多种场景。大多数现有的基础扩散模型主要设计用于文本引导的视觉生成，不支持多模态条件，而多模态条件对于许多视觉编辑任务至关重要。这一局限性使得这些基础扩散模型无法像自然语言处理领域的GPT-4那样，在视觉生成领域充当统一模型。在这项工作中，我们提出了ACE，一个全能创作者与编辑器（All-round Creator and Editor），它在广泛的视觉生成任务中达到了与专家模型相当的性能。为实现这一目标，我们首先引入了一种统一的条件格式，称为长上下文条件单元（Long-context Condition Unit, LCU），并提出了一种新颖的基于Transformer的扩散模型，该模型以LCU作为输入，旨在跨各种生成和编辑任务进行联合训练。此外，我们提出了一种高效的数据收集方法，以解决可用训练数据缺失的问题。该方法涉及通过基于合成或基于聚类的流程获取成对图像，并利用微调的多模态大语言模型为这些图像对提供准确的文本指令。为了全面评估我们模型的性能，我们建立了一个包含多种视觉生成任务的人工标注成对数据的基准。大量实验结果表明了我们的模型在视觉生成领域的优越性。得益于我们模型的全能特性，我们可以轻松构建一个多模态聊天系统，该系统使用单个模型作为后端，响应任何涉及图像创建的交互式请求，从而避免了视觉智能体中通常采用的繁琐流水线。代码和模型将在项目页面上提供：https://ali-vilab.github.io/ace-page/。
英文摘要：
Diffusion models have emerged as a powerful generative technology and have been found to be applicable in various scenarios. Most existing foundational diffusion models are primarily designed for text-guided visual generation and do not support multi-modal conditions, which are essential for many visual editing tasks. This limitation prevents these foundational diffusion models from serving as a unified model in the field of visual generation, like GPT-4 in the natural language processing field. In this work, we propose ACE, an All-round Creator and Editor, which achieves comparable performance compared to those expert models in a wide range of visual generation tasks. To achieve this goal, we first introduce a unified condition format termed Long-context Condition Unit (LCU), and propose a novel Transformer-based diffusion model that uses LCU as input, aiming for joint training across various generation and editing tasks. Furthermore, we propose an efficient data collection approach to address the issue of the absence of available training data. It involves acquiring pairwise images with synthesis-based or clustering-based pipelines and supplying these pairs with accurate textual instructions by leveraging a fine-tuned multi-modal large language model. To comprehensively evaluate the performance of our model, we establish a benchmark of manually annotated pairs data across a variety of visual generation tasks. The extensive experimental results demonstrate the superiority of our model in visual generation fields. Thanks to the all-in-one capabilities of our model, we can easily build a multi-modal chat system that responds to any interactive request for image creation using a single model to serve as the backend, avoiding the cumbersome pipeline typically employed in visual agents. Code and models will be available on the project page: https://ali-vilab.github.io/ace-page/.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
