直面房间里的大象：Visual Prompt Tuning 还是全量微调？
Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning?
AI总结：
本文在19个数据集与任务上分析VPT相较全量微调的适用条件，发现任务目标差异大或数据分布相似时VPT更优，其优势源于保留原特征并新增参数。
摘要：
随着视觉模型规模持续增长，Visual Prompt Tuning（VPT，视觉提示调优）作为一种参数高效的迁移学习技术，因其相较传统全量微调的优越性能而受到关注。然而，有利于 VPT 的条件（“何时”）及其背后的根本原因（“为何”）仍不清楚。本文中，我们在 19 个不同数据集和任务上进行了全面分析。为理解“何时”这一方面，我们从任务目标和数据分布两个维度识别 VPT 表现更有利的场景。我们发现，在以下情况下 VPT 更可取：1）原始任务与下游任务目标之间存在显著差异（例如，从分类转变为计数）；或 2）两个任务的数据分布具有相似性（例如，两者都涉及自然图像）。在探究“为何”这一维度时，结果表明，VPT 的成功不能仅归因于过拟合和优化方面的考虑。VPT 保留原始特征并增加参数的独特方式似乎是一个关键因素。我们的研究深入揭示了 VPT 的机制，并为其最佳使用提供了指导。
英文摘要：
As the scale of vision models continues to grow, the emergence of Visual Prompt Tuning (VPT) as a parameter-efficient transfer learning technique has gained attention due to its superior performance compared to traditional full-finetuning. However, the conditions favoring VPT (the ``when") and the underlying rationale (the ``why") remain unclear. In this paper, we conduct a comprehensive analysis across 19 distinct datasets and tasks. To understand the ``when" aspect, we identify the scenarios where VPT proves favorable by two dimensions: task objectives and data distributions. We find that VPT is preferrable when there is 1) a substantial disparity between the original and the downstream task objectives (e.g., transitioning from classification to counting), or 2) a similarity in data distributions between the two tasks (e.g., both involve natural images). In exploring the ``why" dimension, our results indicate VPT's success cannot be attributed solely to overfitting and optimization considerations. The unique way VPT preserves original features and adds parameters appears to be a pivotal factor. Our study provides insights into VPT's mechanisms, and offers guidance for its optimal utilization.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
