VIP：由多模态大语言模型赋能的通用图像外扩
VIP: Versatile Image Outpainting Empowered by Multimodal Large Language Model
AI总结：
本文针对图像外扩任务通用性和定制化不足的问题，提出基于多模态大语言模型提取文本提示并结合CTS交叉注意力模块的框架，在现成SD模型上微调后显著优于现有最佳方法，支持定制化外扩。
摘要：
本文聚焦于解决图像外扩问题，该任务旨在根据图像的中心内容外推其周围部分。尽管近期研究取得了不错的性能，但通用性和定制化的缺失阻碍了它们在更广泛场景中的实际应用。因此，本工作提出了一种新颖的图像外扩框架，能够根据用户需求定制结果。首先，我们利用多模态大语言模型（MLLM）自动提取并组织给定图像的掩码部分和未掩码部分的对应文本描述。据此，引入所获得的文本提示，使我们的模型具备定制外扩结果的能力。此外，我们精心设计了一个特殊的交叉注意力模块，即中心-整体-周围（CTS）模块，以进一步增强图像特定空间区域与文本提示对应部分之间的交互。需要注意的是，与大多数现有方法不同，我们的方法资源效率极高，因为它仅在现成的稳定扩散（SD）模型上进行轻微微调，而非从头开始训练。最后，在三个常用数据集（即Scenery、Building和WikiArt）上的实验结果表明，我们的模型显著优于现有最佳方法。此外，我们列出了通用的外扩结果以展示其定制能力。
英文摘要：
In this paper, we focus on resolving the problem of image outpainting, which aims to extrapolate the surrounding parts given the center contents of an image. Although recent works have achieved promising performance, the lack of versatility and customization hinders their practical applications in broader scenarios. Therefore, this work presents a novel image outpainting framework that is capable of customizing the results according to the requirement of users. First of all, we take advantage of a Multimodal Large Language Model (MLLM) that automatically extracts and organizes the corresponding textual descriptions of the masked and unmasked part of a given image. Accordingly, the obtained text prompts are introduced to endow our model with the capacity to customize the outpainting results. In addition, a special Cross-Attention module, namely Center-Total-Surrounding (CTS), is elaborately designed to enhance further the the interaction between specific space regions of the image and corresponding parts of the text prompts. Note that unlike most existing methods, our approach is very resource-efficient since it is just slightly fine-tuned on the off-the-shelf stable diffusion (SD) model rather than being trained from scratch. Finally, the experimental results on three commonly used datasets, i.e. Scenery, Building, and WikiArt, demonstrate our model significantly surpasses the SoTA methods. Moreover, versatile outpainting results are listed to show its customized ability.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
