UniVG：用于统一图像生成与编辑的通用扩散模型
UniVG: A Generalist Diffusion Model for Unified Image Generation and Editing
AI总结：
本文提出UniVG，一个以统一条件处理多模态输入、用单一权重集支持多种图像生成与编辑任务的通用扩散模型，通过多任务训练在多个基准上超越专用模型，迈向统一图像生成。
摘要：
文本到图像（T2I）扩散模型在根据用户提示生成视觉上引人注目的图像方面展现了令人印象深刻的结果。在此基础上，各种方法进一步对预训练的T2I模型进行微调以处理特定任务。然而，这需要单独的模型架构、训练设计和多组参数来处理不同的任务。在本文中，我们引入了UniVG，一个通用扩散模型，能够用一组权重支持多种图像生成任务。UniVG将多模态输入视为统一条件，以实现各种下游应用，从T2I生成、修复、基于指令的编辑、身份保持生成、布局引导生成，到深度估计和指代分割。通过对数据混合和多任务训练的全面实证研究，我们提供了关于训练过程和决策的详细见解，这些见解指导了我们的最终设计。例如，我们表明T2I生成和其他任务（如基于指令的编辑）可以共存而无需性能权衡，而辅助任务如深度估计和指代分割则增强了图像编辑。值得注意的是，我们的模型甚至可以在各自基准上超越某些特定任务的模型，这标志着向统一图像生成模型迈出了重要一步。
英文摘要：
Text-to-Image (T2I) diffusion models have shown impressive results in generating visually compelling images following user prompts. Building on this, various methods further fine-tune the pre-trained T2I model for specific tasks. However, this requires separate model architectures, training designs, and multiple parameter sets to handle different tasks. In this paper, we introduce UniVG, a generalist diffusion model capable of supporting a diverse range of image generation tasks with a single set of weights. UniVG treats multi-modal inputs as unified conditions to enable various downstream applications, ranging from T2I generation, inpainting, instruction-based editing, identity-preserving generation, and layout-guided generation, to depth estimation and referring segmentation. Through comprehensive empirical studies on data mixing and multi-task training, we provide detailed insights into the training processes and decisions that inform our final designs. For example, we show that T2I generation and other tasks, such as instruction-based editing, can coexist without performance trade-offs, while auxiliary tasks like depth estimation and referring segmentation enhance image editing. Notably, our model can even outperform some task-specific models on their respective benchmarks, marking a significant step towards a unified image generation model.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
