arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2304.07919cs.CVcs.AI
收藏

视觉语言模型中的思维链提示调优

Chain of Thought Prompt Tuning in Vision Language Models

Peking University(北京大学);MIT(麻省理工学院);BAAI(北京智源人工智能研究院)

Jiaxin Ge, Hongyin Luo, Siyuan Qian, Yulu Gan, Jie Fu, Shanghang Zhang

AI总结:

本文提出一种新颖的思维链提示调优方法,用于视觉语言模型,通过结合视觉和文本嵌入的逐步推理,提升图像分类、检索和问答等任务的泛化与推理能力。

摘要:

语言-图像预训练通过使用自然语言提示对视觉模型进行提示,在零样本和少样本下游任务中展现了有前景的结果。然而,最近的大多数研究仅使用单一提示进行调优,忽略了人类在复杂任务设置中(例如处理来自不熟悉领域的图像时)所进行的逐步认知推理过程。思维链是人类推理过程的一种简单而有效的近似,已被证明对自然语言处理(NLP)任务有用。基于这种认知直觉,我们认为在视觉任务中进行有效推理也是一个重要问题,而思维链可能是解决该问题的一种方案。在这项工作中,我们提出了一种新颖的用于视觉语言建模的思维链提示调优方法。大量实验表明,我们的方法不仅在图像分类任务中具有更好的泛化能力,具有超越单一数据集的更大可迁移性,以及更强的领域泛化性能,而且在需要更多推理能力的图像-文本检索和视觉问答任务中表现也更好。我们是首个成功将结合视觉和文本嵌入的思维链提示进行适配的研究。我们将发布我们的代码。

英文摘要:

Language-Image Pre-training has demonstrated promising results on zero-shot and few-shot downstream tasks by prompting visual models with natural language prompts. However, most recent studies only use a single prompt for tuning, neglecting the inherent step-to-step cognitive reasoning process that humans conduct in complex task settings, for example, when processing images from unfamiliar domains. Chain of Thought is a simple and effective approximation to human reasoning process and has been proven useful for natural language processing (NLP) tasks. Based on this cognitive intuition, we believe that conducting effective reasoning is also an important problem in visual tasks, and a chain of thought could be a solution to this problem. In this work, we propose a novel chain of thought prompt tuning for vision-language modeling. Extensive experiments show that our method not only generalizes better in image classification tasks, has greater transferability beyond a single dataset, and has stronger domain generalization performance, but also performs much better in imagetext retrieval and visual question answering, which require more reasoning capabilities. We are the first to successfully adapt chain-of-thought prompting that combines visual and textual embeddings. We will release our codes

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑