arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2401.15652cs.CV
收藏

基于位置查询与扩散方法的一步式连续多倍图像外扩

Continuous-Multiple Image Outpainting in One-Step via Positional Query and A Diffusion-based Approach

Shanghai Jiao Tong University(上海交通大学);University of Rochester(罗切斯特大学);INF Tech Co., Ltd.(INF科技有限公司);Alibaba Group(阿里巴巴集团)

Shaofeng Zhang, Jinfa Huang, Qiang Zhou, Zhibin Wang, Fan Wang, Jiebo Luo, Junchi Yan

AI总结:

针对图像外扩的倍数限制与多步推理效率问题,提出基于位置查询的扩散模型PQDiff,实现无限制连续倍数单步外扩,在多数据集上取得SOTA精度且推理速度大幅提升。

摘要:

图像外扩旨在生成输入子图像原始边界之外的内容,是内容生成领域的重要任务,但对生成模型而言仍是一个未解决的开放问题。本文从现有文献尚未解决的两个方向推动图像外扩的技术前沿:1)任意连续倍数的无限制外扩;2)单步完成外扩(即使是大扩张倍数)。此外,我们开发的方法不依赖预训练骨干网络,这与此前最优(SOTA)外扩方法通常的要求形成对比。\n任意倍数外扩通过在训练时利用同一图像的随机裁剪视图来捕获任意相对位置信息实现。具体而言,通过将一个视图和位置嵌入作为查询输入,我们可以重建另一个视图。推理时,通过输入锚点图像及其对应的位置嵌入,即可生成任意扩张倍数的图像。这里的单步外扩能力尤其值得关注——此前的方法需要执行N次才能得到基础固定倍数N倍的最终扩张效果。\n我们在公开基准上评估了所提出的方法(因采用扩散生成器作为实现载体,在提出的位置查询(Positional Query)方案下命名为PQDiff),证明其性能优于现有最优方法。具体而言,PQDiff在Scenery(21.512)、Building Facades(25.310)和WikiArts(36.212)数据集上取得了最优FID分数。此外,在2.25倍、5倍和11.7倍外扩设置下,PQDiff的耗时仅为基准最优(SOTA)方法的40.6%、20.3%和10.2%。

英文摘要:

Image outpainting aims to generate the content of an input sub-image beyond its original boundaries. It is an important task in content generation yet remains an open problem for generative models. This paper pushes the technical frontier of image outpainting in two directions that have not been resolved in literature: 1) outpainting with arbitrary and continuous multiples (without restriction), and 2) outpainting in a single step (even for large expansion multiples). Moreover, we develop a method that does not depend on a pre-trained backbone network, which is in contrast commonly required by the previous SOTA outpainting methods. The arbitrary multiple outpainting is achieved by utilizing randomly cropped views from the same image during training to capture arbitrary relative positional information. Specifically, by feeding one view and positional embeddings as queries, we can reconstruct another view. At inference, we generate images with arbitrary expansion multiples by inputting an anchor image and its corresponding positional embeddings. The one-step outpainting ability here is particularly noteworthy in contrast to previous methods that need to be performed for $N$ times to obtain a final multiple which is $N$ times of its basic and fixed multiple. We evaluate the proposed approach (called PQDiff as we adopt a diffusion-based generator as our embodiment, under our proposed \textbf{P}ositional \textbf{Q}uery scheme) on public benchmarks, demonstrating its superior performance over state-of-the-art approaches. Specifically, PQDiff achieves state-of-the-art FID scores on the Scenery (\textbf{21.512}), Building Facades (\textbf{25.310}), and WikiArts (\textbf{36.212}) datasets. Furthermore, under the 2.25x, 5x and 11.7x outpainting settings, PQDiff only takes \textbf{40.6\%}, \textbf{20.3\%} and \textbf{10.2\%} of the time of the benchmark state-of-the-art (SOTA) method.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑