VisualPrompter: 基于视觉反馈的语义感知提示优化用于文本到图像合成
VisualPrompter: Semantic-Aware Prompt Optimization with Visual Feedback for Text-to-Image Synthesis
AI总结：
VisualPrompter通过语义感知的提示优化机制，提升文本到图像合成的对齐性能和内容质量。
摘要：
用户提供的提示与模型偏好提示之间的显著差距对生成高质量图像的文本到图像模型构成了重大挑战，迫使需要提示工程。当前对提示工程的研究可以有效提高生成图像的风格和美学，但往往忽视生成图像与用户描述之间的语义对齐，导致视觉上吸引人但内容上不满意的输出。在本工作中，我们提出了VisualPrompter，一种新的无训练提示工程框架，该框架通过优化用户输入以获得模型偏好的句子。VisualPrompter利用一个自动自我反思模块，该模块可以识别生成图像中缺失的概念，随后是目标特定的提示优化机制，该机制以细粒度的方式修改提示。通过拆解提示、在原子语义层面引入新元素，然后重新组装它们，我们的框架能够在优化过程中保持语义一致性和完整性。广泛的实验展示了VisualPrompter的有效性，该方法在多个文本-图像对齐评估基准上实现了新的最先进的性能。此外，我们的框架具有即插即用的设计，使其能够高度适应各种生成模型。我们的代码可在https://github.com/teheperinko541/VisualPrompter上获得。
英文摘要：
The notable gap between user-provided and model-preferred prompts poses a significant challenge for generating high-quality images with text-to-image models, compelling the need for prompt engineering. Current studies on prompt engineering can effectively enhance the style and aesthetics of generated images. However, they often neglect the semantic alignment between generated images and user descriptions, resulting in visually appealing but content-wise unsatisfying outputs. In this work, we propose VisualPrompter, a novel training-free prompt engineering framework that refines user inputs to model-preferred sentences. VisualPrompter utilizes an automatic self-reflection module that identifies absent concepts in the generated images, followed by a target-specific prompt optimization mechanism that revises the prompts in a fine-grained manner. By deconstructing prompts, introducing new elements at the atomic semantic level, and then reassembling them, our framework is able to maintain semantic consistency and integrity throughout the optimization process. Extensive experiments demonstrate the effectiveness of VisualPrompter, which achieves new state-of-the-art performance on multiple benchmarks for text-image alignment evaluation. Additionally, our framework features a plug-and-play design, making it highly adaptable to various generative models. Our code is available at https://github.com/teheperinko541/VisualPrompter.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
