IP-Prompter：通过动态视觉提示实现无需训练的特定主题图像生成
IP-Prompter: Training-Free Theme-Specific Image Generation via Dynamic Visual Prompting
AI总结：
针对特定主题图像生成中微调成本高、多概念连续性难适配的问题，提出无需训练的IP-Prompter方法，通过动态视觉提示迭代优化参考图像输入，在角色一致性、风格对齐等方面优于现有个性化方法。
摘要：
成长过程中吸引我们的故事与角色塑造了独特的幻想世界，而图像是视觉化体验这些世界的主要媒介。通过使用特定主题数据进行微调来个性化生成模型，已成为文本到图像生成领域的主流方法。然而，与侧重于学习特定对象的对象定制不同，特定主题生成涵盖角色、场景、物体等多元元素。这种多样性也带来了一项关键挑战：如何自适应地生成多角色、多概念且具有连续性的特定主题图像（TSI）。此外，微调方法通常伴随着极高的计算开销、时间成本以及过拟合风险。本文探讨了一个根本性问题：图像生成模型能否像大语言模型利用文本作为上下文一样，直接将图像用作上下文输入？为解决这一问题，我们提出了IP-Prompter，这是一种全新的无需训练的TSI生成方法。IP-Prompter引入了视觉提示机制，可将参考图像整合到生成模型中，用户无需额外训练即可无缝指定目标主题。为进一步优化这一过程，我们提出了动态视觉提示（DVP）机制，通过迭代优化视觉提示来提升生成图像的准确性与质量。我们的方法可应用于多种场景，包括一致性故事生成、角色设计、写实角色生成以及风格引导的图像生成。与当前最先进的个性化方法的对比评估表明，IP-Prompter取得了显著更优的效果，在保持角色身份保留、风格一致性和文本对齐方面表现出色，为特定主题图像生成提供了鲁棒且灵活的解决方案。
英文摘要：
The stories and characters that captivate us as we grow up shape unique fantasy worlds, with images serving as the primary medium for visually experiencing these realms. Personalizing generative models through fine-tuning with theme-specific data has become a prevalent approach in text-to-image generation. However, unlike object customization, which focuses on learning specific objects, theme-specific generation encompasses diverse elements such as characters, scenes, and objects. Such diversity also introduces a key challenge: how to adaptively generate multi-character, multi-concept, and continuous theme-specific images (TSI). Moreover, fine-tuning approaches often come with significant computational overhead, time costs, and risks of overfitting. This paper explores a fundamental question: Can image generation models directly leverage images as contextual input, similarly to how large language models use text as context? To address this, we present IP-Prompter, a novel training-free TSI generation method. IP-Prompter introduces visual prompting, a mechanism that integrates reference images into generative models, allowing users to seamlessly specify the target theme without requiring additional training. To further enhance this process, we propose a Dynamic Visual Prompting (DVP) mechanism, which iteratively optimizes visual prompts to improve the accuracy and quality of generated images. Our approach enables diverse applications, including consistent story generation, character design, realistic character generation, and style-guided image generation. Comparative evaluations against state-of-the-art personalization methods demonstrate that IP-Prompter achieves significantly better results and excels in maintaining character identity preserving, style consistency and text alignment, offering a robust and flexible solution for theme-specific image generation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
