arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2405.06535cs.CVcs.LG
收藏

通过组合并行令牌预测实现可控图像生成

Controllable Image Generation with Composed Parallel Token Prediction

Durham University(杜伦大学)

Jamie Stirling, Noura Al-Moubayed, Chris G. Willcocks, Hubert P. H. Shum

AI总结:

本文提出一种理论指导的离散生成过程组合方法,通过掩码生成(吸收扩散)实现多输入条件的精确组合,提升图像生成的可控性和效率。

摘要:

条件离散生成模型在忠实组合多个输入条件方面存在困难。为解决此问题,我们推导出一种理论指导的组合离散概率生成过程的公式,掩码生成(吸收扩散)是其特殊案例。我们的公式使能够精确指定新的输入条件组合和数量,这些条件不在训练数据中。通过概念加权可以强调或否定个别条件。与VQ-VAE和VQ-GAN丰富的组合性学习词汇相结合,我们的方法在三个数据集上(位置CLEVR、关系CLEVR和FFHQ)相比先前最先进的方法,错误率减少了63.4%。同时,我们的方法在平均绝对FID上实现了-9.58的改进。此外,我们的方法在可比方法上提供了2.3倍到12倍的实时加速,并可轻松应用于开放预训练的离散文本到图像模型,以实现对文本到图像生成的精细控制。

英文摘要:

Conditional discrete generative models struggle to faithfully compose multiple input conditions. To address this, we derive a theoretically-grounded formulation for composing discrete probabilistic generative processes, with masked generation (absorbing diffusion) as a special case. Our formulation enables precise specification of novel combinations and numbers of input conditions that lie outside the training data, with concept weighting enabling emphasis or negation of individual conditions. In synergy with the richly compositional learned vocabulary of VQ-VAE and VQ-GAN, our method attains a $63.4\%$ relative reduction in error rate compared to the previous state-of-the-art, averaged across 3 datasets (positional CLEVR, relational CLEVR and FFHQ), simultaneously obtaining an average absolute FID improvement of $-9.58$. Meanwhile, our method offers a $2.3\times$ to $12\times$ real-time speed-up over comparable methods, and is readily applied to an open pre-trained discrete text-to-image model for fine-grained control of text-to-image generation.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑