LLM引导的基于扩散U-Net交叉注意力图的实例级图像操纵
LLM-guided Instance-level Image Manipulation with Diffusion U-Net Cross-Attention Maps
AI总结：
提出利用LLM、开放词汇检测器及扩散U-Net交叉注意力图实现实例级图像精确操纵，无需微调或辅助掩码，确保图像连贯性。
摘要：
文本到图像合成技术的进步引入了强大的生成模型，能够根据文本提示创建逼真的图像。然而，对图像属性的精确控制仍然具有挑战性，尤其是在实例级别。现有方法通过微调或辅助信息提供一定控制，但往往在灵活性和准确性上存在局限。为解决这些问题，我们提出了一种利用大型语言模型（LLMs）、开放词汇检测器、交叉注意力图以及扩散U-Net中间激活的实例级图像操纵流水线。我们的方法检测提示中提及且存在于生成图像中的对象，无需大量训练或输入掩码即可实现精确操纵。通过整合交叉注意力图，我们的方法在控制对象位置的同时确保操纵图像的连贯性。我们的方法无需微调或掩码、边界框等辅助信息，即可实现实例级别的精确操纵。代码可在 https://github.com/Palandr123/DiffusionU-NetLLM 获取。
英文摘要：
The advancement of text-to-image synthesis has introduced powerful generative models capable of creating realistic images from textual prompts. However, precise control over image attributes remains challenging, especially at the instance level. While existing methods offer some control through fine-tuning or auxiliary information, they often face limitations in flexibility and accuracy. To address these challenges, we propose a pipeline leveraging Large Language Models (LLMs), open-vocabulary detectors, cross-attention maps and intermediate activations of diffusion U-Net for instance-level image manipulation. Our method detects objects mentioned in the prompt and present in the generated image, enabling precise manipulation without extensive training or input masks. By incorporating cross-attention maps, our approach ensures coherence in manipulated images while controlling object positions. Our method enables precise manipulations at the instance level without fine-tuning or auxiliary information such as masks or bounding boxes. Code is available at https://github.com/Palandr123/DiffusionU-NetLLM
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
