BiPrompt：面向视觉与文本去偏的双重视觉语言模型双向提示优化
BiPrompt: Bilateral Prompt Optimization for Visual and Textual Debiasing in Vision-Language Models
AI总结：
BiPrompt通过双重视觉语言模型双向提示优化，同时减轻视觉和文本中的非因果特征依赖，提升模型在分布偏移下的鲁棒性和因果推理能力。
摘要：
视觉语言基础模型如CLIP展现出令人印象深刻的零样本泛化能力，但仍然容易受到跨模态的虚假相关性影响。现有去偏方法往往只针对单一模态（视觉或文本），导致部分鲁棒性和在分布偏移下的不稳定适应。我们提出了一种双重视觉语言模型双向提示优化框架（BiPrompt），在测试时间适应过程中同时减轻两个模态中的非因果特征依赖。在视觉方面，它采用结构化注意力引导擦除来抑制背景激活，并强制因果区域和虚假区域之间的预测一致性。在文本方面，它引入了平衡的提示归一化，这是一种可学习的重新中心机制，将类嵌入向量对齐到一个各向同性的语义空间。共同作用下，这些模块联合最小化虚假线索与预测之间的条件互信息，使模型朝着因果、领域不变的推理方向发展，而无需重新训练或领域监督。在现实世界和合成偏见基准上的广泛评估显示，BiPrompt在平均准确率和最差群体准确率上都比先前的测试时间去偏方法有持续的提升，建立了轻量且有效的可信视觉语言适应路径。
英文摘要：
Vision language foundation models such as CLIP exhibit impressive zero-shot generalization yet remain vulnerable to spurious correlations across visual and textual modalities. Existing debiasing approaches often address a single modality either visual or textual leading to partial robustness and unstable adaptation under distribution shifts. We propose a bilateral prompt optimization framework (BiPrompt) that simultaneously mitigates non-causal feature reliance in both modalities during test-time adaptation. On the visual side, it employs structured attention-guided erasure to suppress background activations and enforce orthogonal prediction consistency between causal and spurious regions. On the textual side, it introduces balanced prompt normalization, a learnable re-centering mechanism that aligns class embeddings toward an isotropic semantic space. Together, these modules jointly minimize conditional mutual information between spurious cues and predictions, steering the model toward causal, domain invariant reasoning without retraining or domain supervision. Extensive evaluations on real-world and synthetic bias benchmarks demonstrate consistent improvements in both average and worst-group accuracies over prior test-time debiasing methods, establishing a lightweight yet effective path toward trustworthy and causally grounded vision-language adaptation.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
