克服视觉-语言模型微调在OOD泛化方面的陷阱
Overcoming the Pitfalls of Vision-Language Model Finetuning for OOD Generalization
AI总结:
本文针对视觉-语言模型微调易过拟合已知类、削弱OOD泛化的问题,提出OGEN,通过类别条件特征生成与自适应自蒸馏正则化决策边界,显著提升未知类泛化性能。
摘要:
现有视觉-语言模型在多种视觉领域和任务上表现出很强的泛化能力。然而,这类模型主要以闭集方式执行零样本识别,因此在设计上难以处理开放领域的视觉概念。最近的一些微调方法(如prompt learning)不仅研究分布内(ID)与分布外(OOD)样本之间的区分,还在ID和OOD准确率上均显示出一定提升。本文首先证明,视觉-语言模型在经过足够长时间的微调但缺乏适当正则化时,往往会过拟合给定数据集中的已知类别,导致未知类别上的性能下降。随后,我们提出一种名为OGEN的新方法来解决这一陷阱,重点是提升微调后模型的OOD泛化能力。具体而言,我们引入一个类别条件特征生成器,仅利用任意未知类别的类名即可合成OOD特征。这些合成特征将提供关于未知类别的有用知识,并在联合优化时帮助正则化ID与OOD数据之间的决策边界。同样重要的是,我们设计了自适应自蒸馏机制,在联合优化过程中正则化特征生成模型,即在不同模型状态之间自适应地迁移知识,以进一步防止过拟合。实验验证了我们的方法在不同设置下均能带来令人信服的OOD泛化性能提升。代码:https://github.com/apple/ml-ogen。
英文摘要:
Existing vision-language models exhibit strong generalization on a variety of visual domains and tasks. However, such models mainly perform zero-shot recognition in a closed-set manner, and thus struggle to handle open-domain visual concepts by design. There are recent finetuning methods, such as prompt learning, that not only study the discrimination between in-distribution (ID) and out-of-distribution (OOD) samples, but also show some improvements in both ID and OOD accuracies. In this paper, we first demonstrate that vision-language models, after long enough finetuning but without proper regularization, tend to overfit the known classes in the given dataset, with degraded performance on unknown classes. Then we propose a novel approach OGEN to address this pitfall, with the main focus on improving the OOD GENeralization of finetuned models. Specifically, a class-conditional feature generator is introduced to synthesize OOD features using just the class name of any unknown class. Such synthesized features will provide useful knowledge about unknowns and help regularize the decision boundary between ID and OOD data when optimized jointly. Equally important is our adaptive self-distillation mechanism to regularize our feature generation model during joint optimization, i.e., adaptively transferring knowledge between model states to further prevent overfitting. Experiments validate that our method yields convincing gains in OOD generalization performance in different settings. Code: https://github.com/apple/ml-ogen.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。