arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2407.00247cs.CV
收藏

基于图像枢轴的提示精炼方法用于文本到图像生成

Prompt Refinement with Image Pivot for Text-to-Image Generation

Tsinghua University(清华大学);Zhongguancun Laboratory(中关村实验室);Renmin University of China(中国人民大学)

Jingtao Zhan, Qingyao Ai, Yiqun Liu, Yingwei Pan, Ting Yao, Jiaxin Mao, Shaoping Ma, Tao Mei

AI总结:

针对文本到图像生成中提示精炼平行语料稀缺的问题,提出PRIP方法,以用户偏好图像的潜在表示为枢轴,将精炼过程分解为图像表示推断和系统语言翻译两个数据丰富任务,显著优于基线并支持零样本迁移。

摘要:

对于文本到图像生成而言,自动将用户提供的自然语言提示精炼为系统所偏好的关键词丰富型提示,对用户体验至关重要。这一提示精炼过程类似于将提示从“用户语言”翻译为“系统语言”。然而,此类平行语料的稀缺使得训练提示精炼模型变得困难。受零样本机器翻译技术的启发,我们提出了基于图像枢轴的提示精炼方法(Prompt Refinement with Image Pivot,PRIP)。PRIP创新性地利用用户偏好图像的潜在表示作为用户语言与系统语言之间的中间“枢轴”。它将精炼过程分解为两个数据丰富的任务:从用户语言推断用户偏好图像的表示,以及随后将图像表示翻译为系统语言。因此,该方法能够利用丰富的数据进行训练。大量实验表明,PRIP显著优于多种基线方法,并能以零样本方式有效迁移到未见过的系统。

英文摘要:

For text-to-image generation, automatically refining user-provided natural language prompts into the keyword-enriched prompts favored by systems is essential for the user experience. Such a prompt refinement process is analogous to translating the prompt from "user languages" into "system languages". However, the scarcity of such parallel corpora makes it difficult to train a prompt refinement model. Inspired by zero-shot machine translation techniques, we introduce Prompt Refinement with Image Pivot (PRIP). PRIP innovatively uses the latent representation of a user-preferred image as an intermediary "pivot" between the user and system languages. It decomposes the refinement process into two data-rich tasks: inferring representations of user-preferred images from user languages and subsequently translating image representations into system languages. Thus, it can leverage abundant data for training. Extensive experiments show that PRIP substantially outperforms a wide range of baselines and effectively transfers to unseen systems in a zero-shot manner.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑