UNIC-Adapter：面向图像生成的统一图像指令适配器与多模态Transformer
UNIC-Adapter: Unified Image-instruction Adapter with Multi-modal Transformer for Image Generation
AI总结：
本文提出UNIC-Adapter，基于多模态扩散Transformer统一多种条件图像生成任务，通过融合条件图像与任务指令并利用旋转位置嵌入增强的交叉注意力机制，实现像素级控制、主体驱动和风格合成等可控生成。
摘要：
近年来，文本到图像生成模型取得了显著进展，尤其是扩散模型能够根据文本描述合成高质量图像。然而，仅使用文本提示时，这些模型往往难以实现对像素级布局、物体外观和全局风格的精确控制。为缓解这一问题，已有工作引入条件图像作为图像生成的辅助输入，从而增强可控性，但通常需要针对不同类型的参考输入定制专用模型。本文探索了一种在单一框架内统一可控生成的新方法。具体而言，我们提出了基于多模态扩散Transformer架构的统一图像指令适配器（UNIC-Adapter），无需多个专用模型即可在多种条件下实现灵活可控的生成。我们的UNIC-Adapter通过同时结合条件图像和任务指令来有效提取多模态指令信息，并通过旋转位置嵌入增强的交叉注意力机制将该信息注入图像生成过程。在像素级空间控制、主体驱动的图像生成以及基于风格图像的图像合成等多种任务上的实验结果表明，我们的UNIC-Adapter在统一可控图像生成方面具有有效性。
英文摘要：
Recently, text-to-image generation models have achieved remarkable advancements, particularly with diffusion models facilitating high-quality image synthesis from textual descriptions. However, these models often struggle with achieving precise control over pixel-level layouts, object appearances, and global styles when using text prompts alone. To mitigate this issue, previous works introduce conditional images as auxiliary inputs for image generation, enhancing control but typically necessitating specialized models tailored to different types of reference inputs. In this paper, we explore a new approach to unify controllable generation within a single framework. Specifically, we propose the unified image-instruction adapter (UNIC-Adapter) built on the Multi-Modal-Diffusion Transformer architecture, to enable flexible and controllable generation across diverse conditions without the need for multiple specialized models. Our UNIC-Adapter effectively extracts multi-modal instruction information by incorporating both conditional images and task instructions, injecting this information into the image generation process through a cross-attention mechanism enhanced by Rotary Position Embedding. Experimental results across a variety of tasks, including pixel-level spatial control, subject-driven image generation, and style-image-based image synthesis, demonstrate the effectiveness of our UNIC-Adapter in unified controllable image generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
