StarFT：通过伪对齐实现零样本模型的鲁棒微调
StarFT: Robust Fine-tuning of Zero-shot Models via Spuriosity Alignment
AI总结：
本文提出 StarFT 框架，通过虚假文本对齐正则化防止零样本模型在微调时学习虚假特征，显著提升了模型的鲁棒泛化能力、零样本组鲁棒性和分类准确率。
摘要：
从数据中学习鲁棒表示通常需要规模，这促成了近期如 CLIP 等零样本模型的成功。然而，当这些模型在其他下游任务（例如较小规模的任务）上进行微调时，所获得的鲁棒性很容易退化。先前的工作通常在领域偏移的背景下解释这一现象，开发了旨在尽可能保留原始领域的微调方法。然而，在不同的背景下，使用有限数据微调的模型也容易学习到对人类而言是虚假的特征，例如背景或纹理。在本文中，我们提出了 StarFT（Spurious Textual Alignment Regularization，虚假文本对齐正则化），这是一种用于微调零样本模型的新框架，通过防止它们学习虚假性来增强鲁棒性。我们引入了一种正则化方法，将注入虚假性标签的输出分布与原始零样本模型对齐，确保模型不会被诱导从这些描述中进一步提取无关特征。我们利用近期的语言模型通过生成突出潜在混淆特征的替代文本描述来获取此类注入虚假性的标签。大量实验验证了 StarFT 的鲁棒泛化能力及其涌现特性：零样本组鲁棒性和改进的零样本分类。值得注意的是，在 Waterbirds 组偏移场景中，StarFT 分别将最差组准确率和平均准确率提升了 14.30% 和 3.02%，而其他鲁棒微调基线甚至表现出性能下降。
英文摘要：
Learning robust representations from data often requires scale, which has led to the success of recent zero-shot models such as CLIP. However, the obtained robustness can easily be deteriorated when these models are fine-tuned on other downstream tasks (e.g., of smaller scales). Previous works often interpret this phenomenon in the context of domain shift, developing fine-tuning methods that aim to preserve the original domain as much as possible. However, in a different context, fine-tuned models with limited data are also prone to learning features that are spurious to humans, such as background or texture. In this paper, we propose StarFT (Spurious Textual Alignment Regularization), a novel framework for fine-tuning zero-shot models to enhance robustness by preventing them from learning spuriosity. We introduce a regularization that aligns the output distribution for spuriosity-injected labels with the original zero-shot model, ensuring that the model is not induced to extract irrelevant features further from these descriptions. We leverage recent language models to get such spuriosity-injected labels by generating alternative textual descriptions that highlight potentially confounding features. Extensive experiments validate the robust generalization of StarFT and its emerging properties: zero-shot group robustness and improved zero-shot classification. Notably, StarFT boosts both worst-group and average accuracy by 14.30% and 3.02%, respectively, in the Waterbirds group shift scenario, where other robust fine-tuning baselines show even degraded performance.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
