我发现了一个隐喻:大型语言模型与扩散模型共同创造视觉隐喻
I Spy a Metaphor: Large Language Models and Diffusion Models Co-Create Visual Metaphors
AI总结:
本文提出从语言隐喻生成视觉隐喻的任务,利用思维链提示的 LLM 生成视觉化细化描述,再由扩散模型成像,并通过人机协作构建数据集与双重评估验证效果。
摘要:
视觉隐喻是强有力的修辞手段,用于通过图像说服他人或传达创造性想法。与语言隐喻类似,视觉隐喻通过象征以及象征之间的并置来隐含地传达意义。本文提出了一项从语言隐喻生成视觉隐喻的新任务。对于 DALL·E 2 等基于扩散模型的文本到图像模型而言,这是一项具有挑战性的任务,因为它需要具备建模隐含意义和组合性的能力。我们建议通过大型语言模型(LLM)与扩散模型之间的协作来解决该任务:采用思维链提示的 Instruct GPT-3(davinci-002)会生成文本,用以表示对语言隐喻的视觉化细化,其中包含隐含意义和相关对象;随后,该文本被用作基于扩散模型的文本到图像模型的输入。借助一个人机协作框架(人类同时与 LLM 和表现最佳的扩散模型交互),我们创建了一个高质量数据集,其中包含面向 1,540 个语言隐喻的 6,476 个视觉隐喻及其相关的视觉化细化内容。专业插画师的评估表明,LLM 与扩散模型协作在这一任务上具有良好前景。为评估我们的人机协作框架的效用以及数据集质量,我们既开展了基于人类的内在评估,也开展了以视觉蕴含作为下游任务的外在评估。
英文摘要:
Visual metaphors are powerful rhetorical devices used to persuade or communicate creative ideas through images. Similar to linguistic metaphors, they convey meaning implicitly through symbolism and juxtaposition of the symbols. We propose a new task of generating visual metaphors from linguistic metaphors. This is a challenging task for diffusion-based text-to-image models, such as DALL$\cdot$E 2, since it requires the ability to model implicit meaning and compositionality. We propose to solve the task through the collaboration between Large Language Models (LLMs) and Diffusion Models: Instruct GPT-3 (davinci-002) with Chain-of-Thought prompting generates text that represents a visual elaboration of the linguistic metaphor containing the implicit meaning and relevant objects, which is then used as input to the diffusion-based text-to-image models.Using a human-AI collaboration framework, where humans interact both with the LLM and the top-performing diffusion model, we create a high-quality dataset containing 6,476 visual metaphors for 1,540 linguistic metaphors and their associated visual elaborations. Evaluation by professional illustrators shows the promise of LLM-Diffusion Model collaboration for this task . To evaluate the utility of our Human-AI collaboration framework and the quality of our dataset, we perform both an intrinsic human-based evaluation and an extrinsic evaluation using visual entailment as a downstream task.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。