arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2303.06594cs.CVcs.AIcs.LG
收藏

ChatGPT提问,BLIP-2回答:对丰富视觉描述的自动提问

ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptions

King Abdullah University of Science and Technology(阿卜杜拉国王科技大学)

Deyao Zhu, Jun Chen, Kilichbek Haydarov, Xiaoqian Shen, Wenxuan Zhang, Mohamed Elhoseiny

摘要:

提出有见地的问题对于获取知识和扩大我们对世界的理解至关重要。然而,在人工智能研究中,提问的重要性在很大程度上被忽视了,因为模型主要是为了回答问题而开发的。随着大语言模型(LLM)(如ChatGPT)的最新进展,我们发现它们在提供适当提示时能够提出高质量的问题。这一发现为开发自动提问系统提供了新的机会。在本文中,我们介绍了ChatCaptioner,一种新的自动提问方法部署在图像描述。在这里,ChatGPT被提示向BLIP-2提出一系列关于图像的信息性问题,BLIP-2是一个强大的视觉问答模型。通过不断从BLIP-2的答案中获取新的视觉信息,ChatCaptioner能够生成更丰富的图像描述。我们对常见的图像标题数据集(如COCO,Conceptual Caption和WikiArt)进行了人类主体评估,并将ChatCaptioner与BLIP-2以及地面实况进行了比较。我们的研究结果表明,ChatCaptioner的标题信息量更大,从人类评估人员那里获得了三倍的投票,以提供最多的图像信息。此外,ChatCaptioner识别图像中的对象比单独使用BLIP-2多53%,这是通过WordNet同义词集匹配测量的。代码可在https://github.com/Vision-CAIR/ChatCaptioner上获得

英文摘要:

Asking insightful questions is crucial for acquiring knowledge and expanding our understanding of the world. However, the importance of questioning has been largely overlooked in AI research, where models have been primarily developed to answer questions. With the recent advancements of large language models (LLMs) like ChatGPT, we discover their capability to ask high-quality questions when provided with a suitable prompt. This discovery presents a new opportunity to develop an automatic questioning system. In this paper, we introduce ChatCaptioner, a novel automatic-questioning method deployed in image captioning. Here, ChatGPT is prompted to ask a series of informative questions about images to BLIP-2, a strong vision question-answering model. By keeping acquiring new visual information from BLIP-2's answers, ChatCaptioner is able to generate more enriched image descriptions. We conduct human-subject evaluations on common image caption datasets such as COCO, Conceptual Caption, and WikiArt, and compare ChatCaptioner with BLIP-2 as well as ground truth. Our results demonstrate that ChatCaptioner's captions are significantly more informative, receiving three times as many votes from human evaluators for providing the most image information. Besides, ChatCaptioner identifies 53% more objects within the image than BLIP-2 alone measured by WordNet synset matching. Code is available at https://github.com/Vision-CAIR/ChatCaptioner

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑