arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2403.19103cs.CVcs.AIcs.CLcs.LG
收藏

面向个性化文本到图像生成的自动化黑盒提示工程

Automated Black-box Prompt Engineering for Personalized Text-to-Image Generation

Carnegie Mellon University(卡内基梅隆大学);University of Pennsylvania(宾夕法尼亚大学);Sony AI(索尼人工智能);Sony Group Corporation(索尼集团);Bosch Center for AI(博世人工智能中心)

Yutong He, Alexander Robey, Naoki Murata, Yiding Jiang, Joshua Nathaniel Williams, George J. Pappas, Hamed Hassani, Yuki Mitsufuji, Ruslan Salakhutdinov, J. Zico Kolter

AI总结:

本文提出PRISM算法,利用大语言模型的上下文学习能力,在仅黑盒访问条件下自动生成可解释、可迁移的文本到图像提示,并在多个主流T2I模型上验证了其有效性。

摘要:

提示工程是控制文本到图像(T2I)生成模型的一种有效但劳动密集的方式。其耗时性和复杂性推动了自动提示生成算法的发展。然而,这些方法往往在跨T2I模型的迁移性上存在困难,需要白盒访问底层模型,或产生不直观的提示。在这项工作中,我们提出了PRISM,一种能够自动生成人类可解释且可迁移的提示的算法,在仅具备对T2I模型的黑盒访问条件下即可有效生成所需概念。受大语言模型(LLM)越狱的启发,PRISM利用LLM的上下文学习能力,迭代地细化基于参考图像构建的候选提示分布。我们的实验证明了PRISM在多个T2I模型(包括Stable Diffusion、DALL-E和Midjourney)上为物体、风格和图像生成准确提示方面的多样性和有效性。

英文摘要:

Prompt engineering is an effective but labor-intensive way to control text-to-image (T2I) generative models. Its time-intensive nature and complexity have spurred the development of algorithms for automated prompt generation. However, these methods often struggle with transferability across T2I models, require white-box access to the underlying model, or produce non-intuitive prompts. In this work, we introduce PRISM, an algorithm that automatically produces human-interpretable and transferable prompts that can effectively generate desired concepts given only black-box access to T2I models. Inspired by large language model (LLM) jailbreaking, PRISM leverages the in-context learning ability of LLMs to iteratively refine the candidate prompt distribution built upon the reference images. Our experiments demonstrate the versatility and effectiveness of PRISM in generating accurate prompts for objects, styles, and images across multiple T2I models, including Stable Diffusion, DALL-E, and Midjourney.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑