发表机构
University of Cambridge; Visa Inc.; University of Manchester（剑桥大学; 维萨公司; 曼彻斯特大学）
机构由 AI 辅助整理，请以论文原文为准。AI 中文总结
本文针对分布偏移下视觉-语言模型的公平性问题，提出了感知公平性的情节式测试时自适应方法FairTPT，通过软提示调优联合优化熵，实现公平性提升并优于现有方法。
AI 中文摘要
视觉-语言模型在多模态理解领域展现出卓越能力，且越来越多地被应用于经济与实际部署约束禁止重新训练或微调的关键场景中。然而，这些模型也可能表现出系统性偏差，对受保护的人口统计群体造成不成比例的影响，而现有的解决这些偏差的方法需要大量的模型重新训练和人口统计属性访问权限。显然需要开发测试时自适应（TTA）方法，以在分布偏移下提升预训练模型的公平性特征。在本文中，我们评估了在子群体偏移下，情节式测试时自适应对CLIP分类公平性的影响，并开发了FairTPT，这是一种新颖的感知公平性的情节式测试时自适应方法，通过软提示调优联合最小化目标边际熵，同时最大化虚假边际熵。我们发现，标准情节式测试时自适应通常会加剧多数群体与少数群体之间的差异；在不降低目标性能的情况下，使模型对虚假属性“失明”本质上具有挑战性；过度“失明”可能会导致灾难性遗忘。这种模型崩溃可通过在线性范围内监控测试时目标损失的变化来预防，同时仍能在反应性数据上实现公平性提升并保持整体性能。FairTPT的表现优于所有最先进的情节式测试时去偏方法，为稳健的测试时自适应奠定了基础，这对在实践中实现公平性至关重要。
英文摘要
Vision-language models have displayed remarkable capabilities in multi-modal understanding and are increasingly used in critical applications where economic and practical deployment constraints prohibit re-training or fine-tuning. However, these models can also exhibit systematic biases that disproportionately affect protected demographic groups and existing approaches to addressing these biases require extensive model retraining and access to demographic attributes. There is a clear need to develop test-time adaptation (TTA) approaches that improve the fairness characteristics of pretrained models under distributional shift. In this paper, we evaluate how episodic TTA affects fairness in CLIP classification under subpopulation shifts and develop FairTPT, a novel fairness-aware episodic TTA method that jointly minimizes target marginal entropy while maximizing spurious marginal entropy through soft-prompt tuning. We find that standard episodic TTA generally exacerbates disparities between majority and minority groups, that blinding a model to spurious attributes without degrading target performance is inherently challenging, and that excessive blinding can lead to catastrophic forgetting. This model collapse can be prevented by monitoring test-time changes in target loss within the linear regime, while still achieving fairness improvements on reactive data and preserving overall performance. FairTPT outperforms all state-of-the-art episodic test-time debiasing methods and establishes a foundation for robust TTA, which is essential for achieving fairness in practice.
