寻找成功的插值：关于锐度在CLIP泛化中的作用
In Search of the Successful Interpolation: On the Role of Sharpness in CLIP Generalization
AI总结：
本文研究CLIP模型鲁棒微调中权重插值的成功条件，发现滞后层的逐层锐度能可靠预测OOD泛化，并通过稀疏化滞后层缓解失效模式。
摘要：
像CLIP这样的零样本模型通常会在目标数据集上进行微调以进一步提高准确率，但这可能会损害分布外（OOD）鲁棒性。鲁棒微调（RFT）通过在零样本模型和微调模型之间进行插值，已被提出来解决这一问题。然而，关于RFT究竟何时能改善OOD误差的理解仍然有限。在这项工作中，我们实证研究了CLIP模型中RFT的鲁棒性，重点关注插值过程中CLIP模型的锐度。首先，我们证明锐度可能无法作为预测现代架构（如CLIP）在OOD数据上泛化能力的可靠指标，这挑战了基础模型中平坦极小值有利于泛化的传统观念。然而，通过考察“滞后层”现象的作用，我们表明，与整体锐度不同，滞后层的逐层锐度能够可靠地反映插值后的CLIP模型在OOD数据上的泛化性能。我们的大量实验表明，逐层锐度与RFT的OOD准确率泛化相关。此外，我们证明通过在滞后层中引入稀疏性，可以缓解RFT中的“失效模式”现象。据我们所知，这是首个研究锐度在CLIP基础模型权重空间插值成功中所起作用的工作。我们的代码可在https://github.com/alirezaabdollahpour/CLIP_Mode_Connectivity获取。
英文摘要：
\textit{Zero-shot} models like CLIP are often fine-tuned on a target dataset to improve its accuracy further, but this can compromise out-of-distribution (OOD) robustness. Robust Fine-Tuning (\texttt{RFT} )~\citep{wortsman2021robust}, which interpolates between the \textit{zero-shot} and \textit{fine-tuned} models, has been proposed to address this issue. However, understanding when \texttt{RFT} actually improves OOD error remains limited. In this work, we empirically investigate the robustness of \texttt{RFT} in CLIP models, with a focus on the \textit{sharpness} of the CLIP model during interpolation. First, we demonstrate that while sharpness may not serve as a reliable indicator for predicting the generalization of modern architectures like CLIP on OOD data, this challenges the conventional belief in the generalization benefits of flat minima in foundation models. However, by examining the role of the \textit{straggler layer} phenomenon, we show that, unlike overall sharpness, the \textit{layer-wise} sharpness of \textit{straggler} layers can reliably capture the generalization performance of interpolated CLIP models on OOD data. Our extensive experiments reveal that \textit{layer-wise} sharpness correlates with generalization in OOD accuracy for \texttt{RFT}. Furthermore, we demonstrate that by inducing sparsity in the \textit{straggler} layers, we can mitigate the \textit{failure mode} phenomenon in \texttt{RFT}. To the best of our knowledge, this is the first work to study the role of sharpness in the \textit{success} of interpolation in the weight space of CLIP foundation models. Our code is available at \url{https://github.com/alirezaabdollahpour/CLIP_Mode_Connectivity}.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
