arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.15328cs.CVcs.AIcs.CLcs.LG
收藏

面向文本到图像生成与评估的可视化编程

Visual Programming for Text-to-Image Generation and Evaluation

UNC Chapel Hill(北卡罗来纳大学教堂山分校)

Jaemin Cho, Abhay Zala, Mohit Bansal

AI总结:

本文提出两个可视化编程框架:VPGen通过语言模型分三步生成图像以增强空间控制,VPEval通过多专家视觉模块实现可解释的T2I评估,两者均优于现有方法。

摘要:

随着大型语言模型在众多领域展现出卓越性能,近期研究已将语言模型(LMs)用作视觉模块的控制器,以处理视觉-语言任务。现有工作主要聚焦于赋予语言模型视觉理解能力,而我们提出了两个新颖的可解释/可说明的可视化编程框架,分别用于文本到图像(T2I)生成与评估。首先,我们提出VPGen,这是一个可解释的逐步T2I生成框架,将T2I生成分解为三个步骤:对象/数量生成、布局生成和图像生成。我们使用一个语言模型来处理前两个步骤(对象/数量生成和布局生成),通过在文本-布局对上对其进行微调来实现。与端到端模型(该任务的主流方法)相比,我们的逐步T2I生成框架提供了更强的空间控制能力。此外,我们利用预训练语言模型的世界知识,克服了以往基于布局引导的T2I工作只能处理预定义对象类别的局限。我们证明,与最先进的T2I生成模型相比,我们的VPGen在对象的数量、空间关系和尺度方面具有更好的控制能力。其次,我们提出VPEval,这是一个基于可视化编程的可解释且可说明的T2I生成评估框架。以往的T2I评估使用单一评分模型,该模型在某些技能上准确但在其他技能上不可靠;与之不同,VPEval生成评估程序,调用一组在不同技能上各有所长的视觉模块,并提供评估结果的可视化和文本化解释。我们的分析表明,与广泛使用的基于单一模型的评估相比,VPEval在技能特定提示和开放式提示上提供了与人类判断更相关的评估结果。我们希望我们的工作能推动T2I模型在可解释/可说明生成与评估方面的未来进展。

英文摘要:

As large language models have demonstrated impressive performance in many domains, recent works have adopted language models (LMs) as controllers of visual modules for vision-and-language tasks. While existing work focuses on equipping LMs with visual understanding, we propose two novel interpretable/explainable visual programming frameworks for text-to-image (T2I) generation and evaluation. First, we introduce VPGen, an interpretable step-by-step T2I generation framework that decomposes T2I generation into three steps: object/count generation, layout generation, and image generation. We employ an LM to handle the first two steps (object/count generation and layout generation), by finetuning it on text-layout pairs. Our step-by-step T2I generation framework provides stronger spatial control than end-to-end models, the dominant approach for this task. Furthermore, we leverage the world knowledge of pretrained LMs, overcoming the limitation of previous layout-guided T2I works that can only handle predefined object classes. We demonstrate that our VPGen has improved control in counts/spatial relations/scales of objects than state-of-the-art T2I generation models. Second, we introduce VPEval, an interpretable and explainable evaluation framework for T2I generation based on visual programming. Unlike previous T2I evaluations with a single scoring model that is accurate in some skills but unreliable in others, VPEval produces evaluation programs that invoke a set of visual modules that are experts in different skills, and also provides visual+textual explanations of the evaluation results. Our analysis shows that VPEval provides a more human-correlated evaluation for skill-specific and open-ended prompts than widely used single model-based evaluation. We hope that our work encourages future progress on interpretable/explainable generation and evaluation for T2I models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑