负责任的文本-视觉对话学习在图像再创作中拒绝人类指令
Accountable Textual-Visual Chat Learns to Reject Human Instructions in Image Re-creation
AI总结:
本文提出两个多模态数据集CLEVR-ATVC和Fruit-ATVC,通过引入规则监督信号,采用两阶段训练从头训练dVAE和自回归transformer,使视觉语言模型在图像再创作中能拒绝不当人类指令并提供解释,增强了文本-视觉生成模型的问责能力。
摘要:
ChatGPT和GPT-4近期的成功引发了人们对多模态对话系统的广泛关注。然而,学术界缺乏能够有效评估视觉语言模型(VLMs)在文本-视觉对话任务中多模态生成能力的数据集。本文通过引入两个新颖的多模态数据集来填补这一空白:合成的CLEVR-ATVC数据集(620K)和手动拍摄的Fruit-ATVC数据集(50K)。这些数据集结合了基于视觉和文本的输入与输出。此外,为了促进多模态系统在拒绝人类请求时的问责能力,类似于基于语言的ChatGPT对话,我们在数据集中引入了特定规则作为监督信号。这使得训练后的VLM在进行视觉和文本推理后能够给出是或否的回答,并附带语言解释以澄清无法执行给定人类指令的原因。我们提出的方法包含一个两阶段训练过程,包括从头训练图像自编码器和自回归transformer。第一阶段采用离散变分自编码器(dVAE)将每张图像压缩为简洁的token,随后与文本token组合成单一数据流。该数据流随后被输入到基于解码器的transformer中,在第二阶段生成视觉再创作和文本反馈。我们对实验结果进行了全面分析,重点关注再创作图像质量、回答准确性,以及模型在面对不确定性和不完美用户查询时的行为。通过我们的探索和发现,旨在为文本-视觉生成模型的问责能力提供有价值的见解。
英文摘要:
The recent success of ChatGPT and GPT-4 has drawn widespread attention to multimodal dialogue systems. However, there is a lack of datasets in the academic community that can effectively evaluate the multimodal generation capabilities of Visual Language Models (VLMs) in textual-visual chat tasks. In this paper, we address this gap by introducing two novel multimodal datasets: the synthetic CLEVR-ATVC dataset (620K) and the manually pictured Fruit-ATVC dataset (50K). These datasets incorporate both visual and text-based inputs and outputs. Furthermore, to facilitate the accountability of multimodal systems in rejecting human requests, similar to language-based ChatGPT conversations, we introduce specific rules as supervisory signals within the datasets. This allows the trained VLM to provide a yes or no answer after engaging in visual and textual reasoning, accompanied by a language explanation to clarify the reasons behind the inability to execute the given human instruction. Our proposed method involves a two-stage training procedure, which includes training the image auto-encoder and the auto-regressive transformer from scratch. The first stage employs a discrete variational autoencoder (dVAE) to compress each image into concise tokens, which are then combined with text tokens into a single data stream. This stream is subsequently fed into the decoder-based transformer to generate visual re-creations and textual feedback in the second stage. We conduct comprehensive analyses of experimental results, focusing on re-created image quality, answer accuracy, and the model's behavior when faced with uncertainty and imperfect user queries. Through our explorations and findings, we aim to contribute valuable insights into the accountability of textual-visual generative models.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。