arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2511.00523cs.CV
收藏

SegDebias:通过分割为基于ViT的CLIP进行测试时偏差缓解

SegDebias: Test-Time Bias Mitigation for ViT-Based CLIP via Segmentation

Fangyu Wu, Yujun Cai

AI总结:

提出一种针对基于ViT的CLIP模型的测试时去偏差方法,利用预训练分割模型隔离目标视觉属性并调整非目标区域,无需额外训练或偏差注释,在组鲁棒性和Attention IoU上优于现有方法。

摘要:

诸如CLIP之类的视觉语言模型在零样本分类中表现出色,但仍然容易受到虚假相关性的影响,即无关的视觉特征会影响预测。现有的去偏差方法通常需要访问训练数据和显式组标签来进行微调或调整嵌入,这限制了它们在现实环境中的实用性。测试时方法试图避免这一约束,但许多方法仍然依赖于对数据集特定偏差的先验知识,限制了它们在开放集环境中的泛化能力。在本研究中,我们提出了一种针对基于ViT的CLIP模型的测试时去偏差方法,该方法不需要额外的训练或偏差注释假设。我们的方法使用预训练的分割模型来隔离目标视觉属性,然后调整非目标区域,使其嵌入与所有类特定的文本提示均匀相似。此过程从混淆视觉区域中移除了意外的偏差信号,同时保留了目标属性。在Waterbirds和CelebA上的实验表明,我们的方法在组鲁棒性指标和Attention IoU方面均优于现有的测试时去偏差方法。这些结果证明了分割引导干预在视觉语言模型中实现可扩展且无需注释的偏差缓解的有效性。

英文摘要:

Vision language models such as CLIP have shown remarkable performance in zero shot classification, but remain susceptible to spurious correlations, where irrelevant visual features influence predictions. Existing debiasing methods often require access to training data and explicit group labels to perform fine-tuning or adjust embeddings, which limits their practicality in real-world settings. Test-time methods attempt to avoid this constraint, but many still depend on prior knowledge of dataset specific biases, limiting their generalizability in open set settings. In this work, we propose a test-time debiasing method for ViT based CLIP models that requires no additional training or assumptions of bias annotations. Our approach uses a pretrained segmentation model to isolate the target visual attribute, then adjusts the non target regions so that their embeddings are uniformly similar to all class specific text prompts. This procedure removes unintended bias signals from confounding visual regions while preserving the target attribute. Experiments on Waterbirds and CelebA show that our method outperforms existing test-time debiasing approaches in both group robustness metrics and Attention IoU. These results demonstrate the effectiveness of segmentation guided interventions for scalable and annotation free bias mitigation in vision language models.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑