arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2401.01952cs.CVcs.AIcs.CL
收藏

Instruct-Imagen:基于多模态指令的图像生成

Instruct-Imagen: Image Generation with Multi-modal Instruction

Google DeepMind(谷歌DeepMind);Google Research(谷歌研究院)

Hexiang Hu, Kelvin C. K. Chan, Yu-Chuan Su, Wenhu Chen, Yandong Li, Kihyuk Sohn, Yang Zhao, Xue Ben, Boqing Gong, William Cohen, Ming-Wei Chang, Xuhui Jia

AI总结:

本文提出instruct-imagen模型,通过引入多模态指令统一异构图像生成任务,并采用两阶段框架微调扩散模型,在域内任务及未见复杂任务上均展现出优异性能与泛化能力。

摘要:

本文提出了instruct-imagen,该模型能够处理异构的图像生成任务,并在未见任务上展现出泛化能力。我们为图像生成引入了*多模态指令*,这是一种能精确表达各种生成意图的任务表示。它使用自然语言融合不同的模态(例如文本、边缘、风格、主体等),从而将丰富的生成意图标准化为统一的格式。随后,我们通过两阶段框架微调预训练的文本到图像扩散模型来构建instruct-imagen。首先,我们使用检索增强训练来调整模型,以增强模型基于外部多模态上下文进行生成的基础能力。随后,我们在需要视觉语言理解的各种图像生成任务(例如主体驱动生成等)上微调调整后的模型,每个任务都配有一条封装任务本质的多模态指令。在各种图像生成数据集上的人工评估表明,instruct-imagen在域内匹配或超越了先前的特定任务模型,并在未见且更复杂的任务上展现出有前景的泛化能力。

英文摘要:

This paper presents instruct-imagen, a model that tackles heterogeneous image generation tasks and generalizes across unseen tasks. We introduce *multi-modal instruction* for image generation, a task representation articulating a range of generation intents with precision. It uses natural language to amalgamate disparate modalities (e.g., text, edge, style, subject, etc.), such that abundant generation intents can be standardized in a uniform format. We then build instruct-imagen by fine-tuning a pre-trained text-to-image diffusion model with a two-stage framework. First, we adapt the model using the retrieval-augmented training, to enhance model's capabilities to ground its generation on external multimodal context. Subsequently, we fine-tune the adapted model on diverse image generation tasks that requires vision-language understanding (e.g., subject-driven generation, etc.), each paired with a multi-modal instruction encapsulating the task's essence. Human evaluation on various image generation datasets reveals that instruct-imagen matches or surpasses prior task-specific models in-domain and demonstrates promising generalization to unseen and more complex tasks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑