我梦绘吾画：通过提示生成连接多模态大语言模型与扩散模型以实现文本引导的多掩码图像修复
I Dream My Painting: Connecting MLLMs and Diffusion Models via Prompt Generation for Text-Guided Multi-Mask Inpainting
AI总结：
本文提出多掩码修复新任务，通过微调多模态大语言模型自动生成多区域提示，并结合修正交叉注意力的Stable Diffusion实现文本引导的精确修复，在绘画数据集上验证了其创意性与准确性。
摘要：
图像修复专注于填补图像中缺失或损坏的区域，使其与周围内容和风格无缝融合。虽然条件扩散模型已被证明在文本引导修复中有效，但我们引入了多掩码修复这一新任务，即使用不同的提示同时修复多个区域。此外，我们为多模态大语言模型（如LLaVA）设计了一种微调流程，以损坏图像作为输入自动生成多掩码提示。这些模型能够为填充掩码区域生成有用且详细的提示建议。生成的提示随后被输入到Stable Diffusion中，该模型通过修正交叉注意力针对多掩码修复问题进行了微调，将提示强制施加到其指定区域进行填充。在WikiArt数字化绘画和Densely Captioned Images数据集上的实验表明，我们的流程能够提供富有创意且准确的修复结果。我们的代码、数据和训练模型可在https://cilabuniba.github.io/i-dream-my-painting获取。
英文摘要：
Inpainting focuses on filling missing or corrupted regions of an image to blend seamlessly with its surrounding content and style. While conditional diffusion models have proven effective for text-guided inpainting, we introduce the novel task of multi-mask inpainting, where multiple regions are simultaneously inpainted using distinct prompts. Furthermore, we design a fine-tuning procedure for multimodal LLMs, such as LLaVA, to generate multi-mask prompts automatically using corrupted images as inputs. These models can generate helpful and detailed prompt suggestions for filling the masked regions. The generated prompts are then fed to Stable Diffusion, which is fine-tuned for the multi-mask inpainting problem using rectified cross-attention, enforcing prompts onto their designated regions for filling. Experiments on digitized paintings from WikiArt and the Densely Captioned Images dataset demonstrate that our pipeline delivers creative and accurate inpainting results. Our code, data, and trained models are available at https://cilabuniba.github.io/i-dream-my-painting.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
