arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2301.12429cs.CV
收藏

通过提示正则化对视觉-语言模型进行去偏微调

Debiased Fine-Tuning for Vision-language Models by Prompt Regularization

NTU(国立新加坡大学)

Beier Zhu, Yulei Niu, Saeil Lee, Minhoe Hur, Hanwang Zhang

AI总结:

提出Prompt Regularization(ProReg)微调范式,利用预训练模型的提示预测作为正则化项,结合自适应权衡权重与KL及交叉熵损失,有效缓解下游任务数据偏差导致的过拟合,在多个分布外基准上取得优异性能。

摘要:

我们提出了一种在下游任务上微调大规模视觉-语言预训练模型的新范式,称为Prompt Regularization(ProReg)。与容易对下游任务数据过拟合的传统微调不同,ProReg利用提示预训练模型得到的预测来正则化微调过程。其动机是:通过提示大模型“a photo of a [CLASS]”,填空答案仅依赖于预训练的百科知识,而独立于通常存在偏差的任务数据分布。具体而言,在微调过程中给定训练样本的预测,我们首先计算提示预测的Kullback-Leibler损失和真实标签的Cross-Entropy损失,然后使用提出的样本级自适应权衡权重将它们结合,该权重自动调整预训练域与下游域之间的迁移。在各种out-of-distribution基准上,我们展示了ProReg相较于传统微调、零样本提示、提示调优及其他state-of-the-art方法的一致强性能。

英文摘要:

We present a new paradigm for fine-tuning large-scale visionlanguage pre-trained models on downstream task, dubbed Prompt Regularization (ProReg). Different from traditional fine-tuning which easily overfits to the downstream task data, ProReg uses the prediction by prompting the pretrained model to regularize the fine-tuning. The motivation is: by prompting the large model "a photo of a [CLASS]", the fil-lin answer is only dependent on the pretraining encyclopedic knowledge while independent of the task data distribution, which is usually biased. Specifically, given a training sample prediction during fine-tuning, we first calculate its KullbackLeibler loss of the prompt prediction and Cross-Entropy loss of the ground-truth label, and then combine them with a proposed sample-wise adaptive trade-off weight, which automatically adjusts the transfer between the pretrained and downstream domains. On various out-of-distribution benchmarks, we show the consistently strong performance of ProReg compared with conventional fine-tuning, zero-shot prompt, prompt tuning, and other state-of-the-art methods.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑