Vision-Language Pre-Training (VLP) has shown promising capabilities to align image and text pairs, facilitating a broad variety of cross-modal learning tasks. However, we observe that VLP models often lack the visual grounding/localization capability which is critical for many downstream tasks such as visual reasoning. In this work, we propose a novel Position-guided Text Prompt (PTP) paradigm to enhance the visual grounding ability of cross-modal models trained with VLP. Specifically, in the VLP phase, PTP divides the image into $N\times N$ blocks, and identifies the objects in each block through the widely used object detector in VLP. It then reformulates the visual grounding task into a fill-in-the-blank problem given a PTP by encouraging the model to predict the objects in the given blocks or regress the blocks of a given object, e.g. filling `P" or ``O" in aPTP ``The block P has a O". This mechanism improves the visual grounding capability of VLP models and thus helps them better handle various downstream tasks. By introducing PTP into several state-of-the-art VLP frameworks, we observe consistently significant improvements across representative cross-modal learning model architectures and several benchmarks, e.g. zero-shot Flickr30K Retrieval (+4.8 in average recall@1) for ViLT \cite{vilt} baseline, and COCO Captioning (+5.3 in CIDEr) for SOTA BLIP \cite{blip} baseline. Moreover, PTP achieves comparable results with object-detector based methods, and much faster inference speed since PTP discards its object detector for inference while the later cannot. Our code and pre-trained weight will be released at \url{https://github.com/sail-sg/ptp}.
Concept-Guided Prompt Learning for Generalization in Vision-Language Models
Harbin Institute of Technology(哈尔滨工业大学);Southern University of Science and Technology(南方科技大学);Carnegie Mellon University(卡内基梅隆大学);Pengcheng Laboratory(鹏城实验室)
Causality-guided Prompt Learning for Vision-language Models via Visual Granulation
State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences(多模态人工智能系统国家重点实验室,自动化研究所,中国科学院);School of Artificial Intelligence, University of Chinese Academy of Sciences(人工智能学院,中国科学院大学)
A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models
University of Oxford(牛津大学);Ludwig Maximilian University of Munich(慕尼黑大学);Google Research(谷歌研究院);Munich Center for Machine Learning(慕尼黑机器学习中心);University of California, Santa Barbara(加州大学圣巴巴拉分校)
GRASP: Guided Region-Aware Sparse Prompting for Adapting MLLMs to Remote Sensing
School of Computing, Kyung Hee University(京畿大学计算机学院);School of Information and Software Engineering, University of Electronic Science and Technology of China(电子科技大学信息与软件工程学院);School of Computer Science and Engineering, University of Electronic Science and Technology of China(电子科技大学计算机科学与工程学院);College of Computer Science and Information Engineering, Harbin Normal University(哈尔滨师范大学计算机科学与信息工程学院);College of Professional and Continuing Education, The Hong Kong Polytechnic University(香港理工大学专业及继续教育学院);School of Computer Science and Technology, Tongji University(同济大学计算机科学与技术学院)
In the Eye of MLLM: Benchmarking Egocentric Video Intent Understanding with Gaze-Guided Prompting
State Key Laboratory of VR Technology and Systems, School of CSE, Beihang University(虚拟现实技术与系统国家重点实验室,北京航空航天大学计算机科学与工程学院);College of AI, Tsinghua University(清华大学人工智能学院)
Guided Prompt Evolution for Vision-Language Models Adaptation
Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院,清华大学);Sun Yat-sen University(中山大学);Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))
Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learning
Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所);University of Chinese Academy of Sciences(中国科学院大学);Peng Cheng Laboratory(鹏城实验室);Wuhan AI Research(武汉人工智能研究院)
Prompt Tuning with Soft Context Sharing for Vision-Language Models
State Key Laboratory of Multimodal Artificial Intelligence Systems, CASIA(中国科学院自动化研究所多模态人工智能系统国家重点实验室);Engineering Laboratory for Intelligent Industrial Vision, CASIA(中国科学院自动化研究所智能工业视觉工程实验室);Research Center of Aerospace Information, CASIA(中国科学院自动化研究所航天信息研究中心)
PLOT: Prompt Learning with Optimal Transport for Vision-Language Models
Carnegie Mellon University(卡内基梅隆大学);Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学);Tsinghua University(清华大学);New York University(纽约大学)
MoPD: Mixture-of-Prompts Distillation for Vision-Language Models
Department of Computer Science and Engineering, Southern University of Science and Technology(计算机科学与工程系,南方科技大学);School of Mathematical and Computer Sciences, University of Adelaide(数学与计算机科学学院,阿德莱德大学)
LDEPrompt: Layer-importance guided Dual Expandable Prompt Pool for Pre-trained Model-based Class-Incremental Learning
School of Information and Communication Engineering, BUPT(北京邮电大学信息与通信工程学院);Engineering Research Center for Information Network, Ministry of Education(教育部信息网络工程研究中心)
MedSAM-U: Uncertainty-Guided Auto Multi-Prompt Adaptation for Reliable MedSAM
College of Computer Science, Sichuan University(四川大学计算机学院);West China Hospital, Sichuan University(四川大学华西医院);Institute of High Performance Computing (IHPC), Agency for Science, Technology and Research (A*STAR)(新加坡科技研究局高性能计算研究所)
Prompt-guided Scene Generation for 3D Zero-Shot Learning
The University of New South Wales(新南威尔士大学);Australian National University(澳大利亚国立大学);North South University(北南大学);University of Technology Sydney(悉尼科技大学);Commonwealth Scientific and Industrial Research Organisation(联邦科学与工业研究组织)