arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.11147cs.CVcs.AI
收藏

UniControl:面向真实场景可控视觉生成的统一扩散模型

UniControl: A Unified Diffusion Model for Controllable Visual Generation In the Wild

Salesforce AI Research(赛富时人工智能研究院);Northeastern University(东北大学)

Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang, Yingbo Zhou, Huan Wang, Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, Stefano Ermon, Yun Fu, Ran Xu

AI总结:

本文提出 UniControl,通过任务感知 HyperNet 调制预训练扩散模型,在单一框架中统一九类 C2I 任务,并支持语言提示与像素级视觉控制,零样本能力优于相当规模的单任务方法。

摘要:

在交互式 AI 系统的设计中,实现机器自主性与人类控制往往代表着相互分歧的目标。Stable Diffusion 等视觉生成基础模型在协调这些目标方面展现出前景,尤其是在使用任意语言作为提示时。然而,它们在生成具有空间、结构或几何控制的图像时通常存在不足。如何整合此类控制,使单一统一模型能够适应各种视觉条件,仍是一个尚未解决的挑战。为此,我们提出 UniControl,这是一种新的生成基础模型,在单一框架内整合了大量可控的条件到图像(C2I)任务,同时仍支持任意语言提示。UniControl 能够实现像素级精确的图像生成,其中视觉条件主要影响生成的结构,语言提示则引导风格和上下文。为使 UniControl 具备处理多样视觉条件的能力,我们增强了预训练的文本到图像扩散模型,并引入任务感知 HyperNet 来调制扩散模型,从而同时适应不同的 C2I 任务。UniControl 在九个独特的 C2I 任务上进行训练,并对未见过的视觉条件表现出令人印象深刻的零样本生成能力。实验结果表明,UniControl 往往超过模型规模相当的单任务控制方法的性能。这种控制多功能性使 UniControl 成为可控视觉生成领域的一项重要进展。

英文摘要:

Achieving machine autonomy and human control often represent divergent objectives in the design of interactive AI systems. Visual generative foundation models such as Stable Diffusion show promise in navigating these goals, especially when prompted with arbitrary languages. However, they often fall short in generating images with spatial, structural, or geometric controls. The integration of such controls, which can accommodate various visual conditions in a single unified model, remains an unaddressed challenge. In response, we introduce UniControl, a new generative foundation model that consolidates a wide array of controllable condition-to-image (C2I) tasks within a singular framework, while still allowing for arbitrary language prompts. UniControl enables pixel-level-precise image generation, where visual conditions primarily influence the generated structures and language prompts guide the style and context. To equip UniControl with the capacity to handle diverse visual conditions, we augment pretrained text-to-image diffusion models and introduce a task-aware HyperNet to modulate the diffusion models, enabling the adaptation to different C2I tasks simultaneously. Trained on nine unique C2I tasks, UniControl demonstrates impressive zero-shot generation abilities with unseen visual conditions. Experimental results show that UniControl often surpasses the performance of single-task-controlled methods of comparable model sizes. This control versatility positions UniControl as a significant advancement in the realm of controllable visual generation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑