arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2312.02150cs.CV
收藏

Readout Guidance:从扩散特征学习控制

Readout Guidance: Learning Control from Diffusion Features

Google Research(谷歌研究院);UC Berkeley(加州大学伯克利分校)

Grace Luo, Trevor Darrell, Oliver Wang, Dan B Goldman, Aleksander Holynski

AI总结:

本文提出 Readout Guidance,通过轻量读出头从冻结扩散模型特征中提取单图或多图信号,并反传梯度引导采样,以更少参数和训练样本统一实现多种图像条件控制。

摘要:

我们提出了 Readout Guidance,这是一种利用学习到的信号控制文本到图像扩散模型的方法。Readout Guidance 使用 readout heads(读出头),即经过训练的轻量级网络,用于在每个时间步从预训练且冻结的扩散模型特征中提取信号。这些读出结果可以编码单幅图像属性,例如姿态、深度和边缘;也可以编码关联多幅图像的高阶属性,例如对应关系和外观相似性。此外,通过将读出估计与用户定义的目标进行比较,并通过读出头反向传播梯度,这些估计可用于引导采样过程。与以往的条件生成方法相比,Readout Guidance 所需的新增参数和训练样本显著更少,并提供了一种便捷简单的方案,可在单一框架下,用单一架构和采样流程复现不同形式的条件控制。我们在基于拖拽的操控、身份一致生成以及空间对齐控制等应用中展示了这些优势。项目主页:https://readout-guidance.github.io。

英文摘要:

We present Readout Guidance, a method for controlling text-to-image diffusion models with learned signals. Readout Guidance uses readout heads, lightweight networks trained to extract signals from the features of a pre-trained, frozen diffusion model at every timestep. These readouts can encode single-image properties, such as pose, depth, and edges; or higher-order properties that relate multiple images, such as correspondence and appearance similarity. Furthermore, by comparing the readout estimates to a user-defined target, and back-propagating the gradient through the readout head, these estimates can be used to guide the sampling process. Compared to prior methods for conditional generation, Readout Guidance requires significantly fewer added parameters and training samples, and offers a convenient and simple recipe for reproducing different forms of conditional control under a single framework, with a single architecture and sampling procedure. We showcase these benefits in the applications of drag-based manipulation, identity-consistent generation, and spatially aligned control. Project page: https://readout-guidance.github.io.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑