arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2307.12980cs.CV
收藏

视觉-语言基础模型提示工程的系统性综述

A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models

University of Oxford(牛津大学);Ludwig Maximilian University of Munich(慕尼黑大学);Google Research(谷歌研究院);Munich Center for Machine Learning(慕尼黑机器学习中心);University of California, Santa Barbara(加州大学圣巴巴拉分校)

Jindong Gu, Zhen Han, Shuo Chen, Ahmad Beirami, Bailan He, Gengyuan Zhang, Ruotong Liao, Yao Qin, Volker Tresp, Philip Torr

AI总结:

本文针对当前视觉-语言模型提示工程缺乏系统梳理的问题,综述三类主流视觉-语言模型的提示方法、应用及伦理问题,对比不同模态模型的提示差异,并指明未来研究方向。

摘要:

提示工程是一种为大型预训练模型添加任务特定提示(即prompts)以适配新任务的技术。提示可手动构建为自然语言指令,也可自动生成为自然语言指令或向量表示。提示工程支持仅依托提示完成预测而无需更新模型参数,能让大型预训练模型更易落地到实际任务中。过去数年,提示工程在自然语言处理领域已得到充分研究。近期,它也在视觉-语言建模领域成为研究热点,但目前尚缺乏针对预训练视觉-语言模型提示工程的系统性梳理。本文旨在全面综述三类视觉-语言模型上提示工程的前沿研究:多模态到文本生成模型(如Flamingo)、图文匹配模型(如CLIP)以及文本到图像生成模型(如Stable Diffusion)。针对每类模型,本文分别梳理并讨论了模型简介、提示方法、基于提示的应用,以及对应的责任与诚信问题。此外,本文还探讨了视觉-语言模型、语言模型与视觉模型的提示技术之间的异同。最后总结了该领域的挑战、未来方向与研究机遇,以推动相关后续研究。

英文摘要:

Prompt engineering is a technique that involves augmenting a large pre-trained model with task-specific hints, known as prompts, to adapt the model to new tasks. Prompts can be created manually as natural language instructions or generated automatically as either natural language instructions or vector representations. Prompt engineering enables the ability to perform predictions based solely on prompts without updating model parameters, and the easier application of large pre-trained models in real-world tasks. In past years, Prompt engineering has been well-studied in natural language processing. Recently, it has also been intensively studied in vision-language modeling. However, there is currently a lack of a systematic overview of prompt engineering on pre-trained vision-language models. This paper aims to provide a comprehensive survey of cutting-edge research in prompt engineering on three types of vision-language models: multimodal-to-text generation models (e.g. Flamingo), image-text matching models (e.g. CLIP), and text-to-image generation models (e.g. Stable Diffusion). For each type of model, a brief model summary, prompting methods, prompting-based applications, and the corresponding responsibility and integrity issues are summarized and discussed. Furthermore, the commonalities and differences between prompting on vision-language models, language models, and vision models are also discussed. The challenges, future directions, and research opportunities are summarized to foster future research on this topic.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑