A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models
University of Oxford(牛津大学);Ludwig Maximilian University of Munich(慕尼黑大学);Google Research(谷歌研究院);Munich Center for Machine Learning(慕尼黑机器学习中心);University of California, Santa Barbara(加州大学圣巴巴拉分校)
Jindong Gu, Zhen Han, Shuo Chen, Ahmad Beirami, Bailan He, Gengyuan Zhang, Ruotong Liao, Yao Qin, Volker Tresp, Philip Torr
Prompt engineering is a technique that involves augmenting a large pre-trained model with task-specific hints, known as prompts, to adapt the model to new tasks. Prompts can be created manually as natural language instructions or generated automatically as either natural language instructions or vector representations. Prompt engineering enables the ability to perform predictions based solely on prompts without updating model parameters, and the easier application of large pre-trained models in real-world tasks. In past years, Prompt engineering has been well-studied in natural language processing. Recently, it has also been intensively studied in vision-language modeling. However, there is currently a lack of a systematic overview of prompt engineering on pre-trained vision-language models. This paper aims to provide a comprehensive survey of cutting-edge research in prompt engineering on three types of vision-language models: multimodal-to-text generation models (e.g. Flamingo), image-text matching models (e.g. CLIP), and text-to-image generation models (e.g. Stable Diffusion). For each type of model, a brief model summary, prompting methods, prompting-based applications, and the corresponding responsibility and integrity issues are summarized and discussed. Furthermore, the commonalities and differences between prompting on vision-language models, language models, and vision models are also discussed. The challenges, future directions, and research opportunities are summarized to foster future research on this topic.
Domain Generalization for Face Anti-spoofing via Content-aware Composite Prompt Engineering
School of Computer Science, Hefei University of Technology(合肥工业大学计算机学院);State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), Institute of Automation, Chinese Academy of Sciences (CASIA)(中国科学院自动化研究所);School of Electronic Information Engineering, Taiyuan University of Technology(太原理工大学电子信息工程学院);School of Computer Science and Engineering, Faculty of Innovation Engineering, Macau University of Science and Technology(澳门科技大学创新工程学院计算机科学与工程学院);School of Cyber Science and Engineering, Wuhan University(武汉大学网络空间安全学院)
Review of Large Vision Models and Visual Prompt Engineering
Northwestern Polytechnical University(西北工业大学);The University of Georgia(佐治亚大学);City University of Hong Kong(香港城市大学);Shaanxi Normal University(陕西师范大学);University of Electronic Science and Technology of China(电子科技大学);The University of Texas at Arlington(德克萨斯大学阿灵顿分校);Massachusetts General Hospital(麻省总医院);Harvard Medical School(哈佛医学院)
A Framework for Critical Evaluation of Text-to-Image Models: Integrating Art Historical Analysis, Artistic Exploration, and Critical Prompt Engineering
School of Fine Arts, University of Ioannina(约阿尼纳大学美术学院)
Prompt me a Dataset: An investigation of text-image prompting for historical image dataset creation using foundation models
Max Planck Institute for the History of Science(马克斯·普朗克科学史研究所);BIFOLD – Berlin Institute for the Foundations of Learning and Data(柏林学习与数据基础研究所);Tel-Aviv University(特拉维夫大学);Technische Universität Berlin(柏林工业大学)