表征层面的反事实校准用于去偏零样本识别
Representation-Level Counterfactual Calibration for Debiased Zero-Shot Recognition
AI总结：
本文将视觉-语言模型中的对象-上下文捷径问题重构为因果推断，在CLIP表征空间合成反事实嵌入并估计总直接效应以减去背景激活，无需重训即显著提升零样本识别的去偏效果与准确率。
摘要：
对象-上下文捷径在视觉-语言模型中仍是一个持续的挑战，当测试时场景不同于熟悉的训练共现时，会破坏零样本可靠性。我们将此问题重新构建为因果推断问题并提问：如果对象出现在不同的环境中，预测是否仍会保持不变？为了在推理时回答此问题，我们在CLIP的表征空间内估计对象与背景的期望，并通过将对象特征与从外部数据集、批次邻居或文本派生描述中采样的多种替代上下文重新组合，合成反事实嵌入。通过估计总直接效应并模拟干预，我们进一步减去仅背景的激活，在保留有益的对象-上下文交互的同时减轻幻觉分数。无需重训或提示设计，我们的方法在上下文敏感的基准上显著提升了最差组和平均准确率，确立了新的零样本最先进水平。除性能外，我们的框架提供了一种轻量级的表征层面反事实方法，为去偏且可靠的多模态推理提供了实用的因果途径。
英文摘要：
Object-context shortcuts remain a persistent challenge in vision-language models, undermining zero-shot reliability when test-time scenes differ from familiar training co-occurrences. We recast this issue as a causal inference problem and ask: Would the prediction remain if the object appeared in a different environment? To answer this at inference time, we estimate object and background expectations within CLIP's representation space, and synthesize counterfactual embeddings by recombining object features with diverse alternative contexts sampled from external datasets, batch neighbors, or text-derived descriptions. By estimating the Total Direct Effect and simulating intervention, we further subtract background-only activation, preserving beneficial object-context interactions while mitigating hallucinated scores. Without retraining or prompt design, our method substantially improves both worst-group and average accuracy on context-sensitive benchmarks, establishing a new zero-shot state of the art. Beyond performance, our framework provides a lightweight representation-level counterfactual approach, offering a practical causal avenue for debiased and reliable multimodal reasoning.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
