图像生成模型的提示恢复:离散优化器的比较研究
Prompt Recovery for Image Generation Models: A Comparative Study of Discrete Optimizers
AI总结:
本研究首次直接比较了GCG、PEZ、Random Search、AutoDAN和BLIP2等离散优化技术及图像描述器在图像生成模型提示反演中的表现,发现依赖CLIP相似度作为代理指标效果不佳,且训练良好的描述器往往优于离散优化器。
摘要:
仅基于生成图像来恢复图像生成模型的自然语言提示是一个困难的离散优化问题。在本研究中,我们首次对近期用于提示反演问题的离散优化技术进行了直接对比。我们评估了 Greedy Coordinate Gradients (GCG)、PEZ、Random Search、AutoDAN 和 BLIP2 的图像描述器,考察指标涵盖反演提示的质量以及由反演提示生成的图像质量。我们发现,将反演提示与真实图像间的 CLIP 相似度作为代理指标,无法有效反映真实图像与由反演提示生成图像之间的相似度。尽管离散优化器能有效最小化其目标函数,但仅使用训练良好的描述器的输出,通常能生成与原始提示生成图像更相似的图像。
英文摘要:
Recovering natural language prompts for image generation models, solely based on the generated images is a difficult discrete optimization problem. In this work, we present the first head-to-head comparison of recent discrete optimization techniques for the problem of prompt inversion. We evaluate Greedy Coordinate Gradients (GCG), PEZ , Random Search, AutoDAN and BLIP2's image captioner across various evaluation metrics related to the quality of inverted prompts and the quality of the images generated by the inverted prompts. We find that focusing on the CLIP similarity between the inverted prompts and the ground truth image acts as a poor proxy for the similarity between ground truth image and the image generated by the inverted prompts. While the discrete optimizers effectively minimize their objectives, simply using responses from a well-trained captioner often leads to generated images that more closely resemble those produced by the original prompts.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。