预训练视觉-语言模型上适应方法鲁棒性的基准测试
Benchmarking Robustness of Adaptation Methods on Pre-trained Vision-Language Models
AI总结：
该研究评估了11种适应方法在多模态损坏下的鲁棒性，发现适应方法对文本损坏更敏感，全量微调未必最优，增加数据和参数反而可能降低鲁棒性。
摘要：
为了提升预训练视觉-语言模型在特定领域的性能，研究者提出了多种适应方法，如LoRA、prompts和adapters。然而，这些适应方法针对分布偏移的鲁棒性尚未得到研究。在本研究中，我们评估了11种广泛使用的适应方法在多模态损坏下跨4个视觉-语言数据集的鲁棒性。具体而言，我们引入了7个基准数据集，包含96种视觉损坏和87种文本损坏，以研究不同适应方法的鲁棒性、可用适应样本的影响，以及适应过程中可训练参数规模的影响。我们的分析表明：1）适应方法对文本损坏比视觉损坏更敏感。2）全量微调并不能始终提供最高的鲁棒性；相反，adapters能够在具有可比的干净性能下实现更好的鲁棒性。3）与预期相反，我们的发现表明，增加适应数据和参数的数量并不能保证增强鲁棒性；反而会导致更低的鲁棒性。我们希望本研究能有助于未来鲁棒多模态适应方法开发的研究。本研究使用的基准、代码和数据集可在 https://adarobustness.github.io 访问。
英文摘要：
Various adaptation methods, such as LoRA, prompts, and adapters, have been proposed to enhance the performance of pre-trained vision-language models in specific domains. The robustness of these adaptation methods against distribution shifts have not been studied. In this study, we assess the robustness of 11 widely-used adaptation methods across 4 vision-language datasets under multimodal corruptions. Concretely, we introduce 7 benchmark datasets, including 96 visual and 87 textual corruptions, to investigate the robustness of different adaptation methods, the impact of available adaptation examples, and the influence of trainable parameter size during adaptation. Our analysis reveals that: 1) Adaptation methods are more sensitive to text corruptions than visual corruptions. 2) Full fine-tuning does not consistently provide the highest robustness; instead, adapters can achieve better robustness with comparable clean performance. 3) Contrary to expectations, our findings indicate that increasing the number of adaptation data and parameters does not guarantee enhanced robustness; instead it results in even lower robustness. We hope this study could benefit future research in the development of robust multimodal adaptation methods. The benchmark, code, and dataset used in this study can be accessed at https://adarobustness.github.io .
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
