PixWizard：具备开放语言指令的多功能图像到图像视觉助手
PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructions
AI总结：
PixWizard是一个基于扩散变换器的多功能图像到图像助手，通过统一框架和指令微调数据集，支持多种视觉任务，并展现出强大的生成与泛化能力。
摘要：
本文介绍了一个多功能的图像到图像视觉助手PixWizard，它旨在基于自由形式的语言指令进行图像生成、操作和转换。为此，我们将多种视觉任务统一到一个图像-文本到图像的生成框架中，并精心构建了一个Omni Pixel-to-Pixel指令微调数据集。通过用自然语言构造详细的指令模板，我们全面涵盖了各种不同的视觉任务，例如文本到图像生成、图像恢复、图像接地、密集图像预测、图像编辑、可控生成、内绘/外绘等等。此外，我们采用扩散变换器（DiT）作为基础模型，并通过灵活的任意分辨率机制扩展其能力，使模型能够根据输入图像的宽高比动态处理图像，与人类的感知过程紧密对齐。该模型还集成了结构感知和语义感知的指导，以促进输入图像信息的有效融合。我们的实验表明，PixWizard不仅对各种分辨率的图像展现出令人印象深刻的生成和理解能力，而且在面对未见过的任务和人类指令时也表现出良好的泛化能力。代码及相关资源可在https://github.com/AFeng-x/PixWizard获取。
英文摘要：
This paper presents a versatile image-to-image visual assistant, PixWizard, designed for image generation, manipulation, and translation based on free-from language instructions. To this end, we tackle a variety of vision tasks into a unified image-text-to-image generation framework and curate an Omni Pixel-to-Pixel Instruction-Tuning Dataset. By constructing detailed instruction templates in natural language, we comprehensively include a large set of diverse vision tasks such as text-to-image generation, image restoration, image grounding, dense image prediction, image editing, controllable generation, inpainting/outpainting, and more. Furthermore, we adopt Diffusion Transformers (DiT) as our foundation model and extend its capabilities with a flexible any resolution mechanism, enabling the model to dynamically process images based on the aspect ratio of the input, closely aligning with human perceptual processes. The model also incorporates structure-aware and semantic-aware guidance to facilitate effective fusion of information from the input image. Our experiments demonstrate that PixWizard not only shows impressive generative and understanding abilities for images with diverse resolutions but also exhibits promising generalization capabilities with unseen tasks and human instructions. The code and related resources are available at https://github.com/AFeng-x/PixWizard
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
