面向视觉-语言模型的鲁棒提示研究
Towards Robust Prompts on Vision-Language Models
AI总结：
本文针对视觉-语言模型在分布偏移下的鲁棒性问题，定义了基类与新类别两类鲁棒性，并提出融合多尺度图像特征的鲁棒提示学习方法，在六个基准上验证其同时提升两类鲁棒性的有效性。
摘要：
随着能够进行上下文学习和基于提示学习的视觉-语言模型（VLMs）的出现，我们如何设计能够在分布偏移下鲁棒泛化、并可用于提示支持集之外的新类别的提示方法？在这项工作中，我们首先定义了视觉-语言模型上两类分布偏移鲁棒性，即基类（提示支持集中包含的类别）鲁棒性和新类别鲁棒性。然后，我们研究了现有上下文学习和提示学习方法的鲁棒性，发现提示学习在基类测试图像上表现鲁棒，但在新类别图像上泛化不佳。我们提出通过将多尺度图像特征集成到提示中来实现鲁棒提示学习，从而同时提升两类鲁棒性。我们在六个基准上进行了全面实验以研究所定义的鲁棒性，并验证了所提方法的有效性。
英文摘要：
With the advent of vision-language models (VLMs) that can perform in-context and prompt-based learning, how can we design prompting approaches that robustly generalize to distribution shift and can be used on novel classes outside the support set of the prompts? In this work, we first define two types of robustness to distribution shift on VLMs, namely, robustness on base classes (the classes included in the support set of prompts) and robustness on novel classes. Then, we study the robustness of existing in-context learning and prompt learning approaches, where we find that prompt learning performs robustly on test images from base classes, while it does not generalize well on images from novel classes. We propose robust prompt learning by integrating multiple-scale image features into the prompt, which improves both types of robustness. Comprehensive experiments are conducted to study the defined robustness on six benchmarks and show the effectiveness of our proposal.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
