量化任务错位以指导 PEFT：CLIP 中领域泛化与灾难性遗忘的探索
Quantified Task Misalignment to Inform PEFT: An Exploration of Domain Generalization and Catastrophic Forgetting in CLIP
AI总结：
本文分析了 CLIP 模型中任务难度与参数高效微调方法性能的关系，发现零样本嵌入的轮廓系数是更好的难度指标，并提出 A-CLIP 方法在领域泛化与灾难性遗忘间取得平衡。
摘要：
基础模型通常被作为通用模型呈现，在众多任务上表现良好。微调这些模型，即使在有限数据上，也能在任务特定性能上提供额外提升，但往往以牺牲更广泛的泛化能力为代价，这种效应被称为灾难性遗忘。本文通过领域泛化和灾难性遗忘的视角，分析了 CLIP 模型中任务难度与几种简单参数高效微调方法性能之间的关系。我们提供证据表明，零样本图像和文本嵌入的轮廓系数是比正确图像/标签嵌入的平均余弦相似度更好的任务难度衡量标准，并讨论了任务难度、微调方法、领域泛化和灾难性遗忘之间可观察到的关系。此外，跨任务和性能指标的平均结果表明，一种仅训练注意力权重子集的简化方法（我们称之为 A-CLIP）在领域泛化和灾难性遗忘之间取得了平衡。
英文摘要：
Foundations models are presented as generalists that often perform well over a myriad of tasks. Fine-tuning these models, even on limited data, provides an additional boost in task-specific performance but often at the cost of their wider generalization, an effect termed catastrophic forgetting. In this paper, we analyze the relation between task difficulty in the CLIP model and the performance of several simple parameter-efficient fine-tuning methods through the lens of domain generalization and catastrophic forgetting. We provide evidence that the silhouette score of the zero-shot image and text embeddings is a better measure of task difficulty than the average cosine similarity of correct image/label embeddings, and discuss observable relationships between task difficulty, fine-tuning method, domain generalization, and catastrophic forgetting. Additionally, the averaged results across tasks and performance measures demonstrate that a simplified method that trains only a subset of attention weights, which we call A-CLIP, yields a balance between domain generalization and catastrophic forgetting.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
