图中标签真的在说些什么？用于视觉语言预训练中组合数据选择的反事实短语干预
What Does the Caption Really Say? Counterfactual Phrase Intervention for Compositional Data Selection in Vision-Language Pretraining
AI总结：
本文研究了在视觉语言预训练中如何通过反事实短语干预来改进组合数据选择，提出了CPI方法以解决现有方法中全局过滤信号失效的问题，从而提升模型在关系识别任务上的表现。
摘要：
CLIP风格的对比预训练通常通过样本级过滤信号来收集网络级图像-文本对，通常基于对级对齐。我们证明这种信号饱和：一旦粗略不匹配被移除，更严格的全局过滤不再跟踪由保留标签提供的组合监督。原因在于结构问题 - 全局评分混淆了对是否广泛合理与是否个别对象、属性和关系短语在标签中实质性支持图像-文本匹配。后者是组合泛化所需，但对级过滤器对此无能为力。我们通过反事实短语干预（CPI），一种短语级整理框架，将受控的非正式令牌替换转换为图像条件的短语敏感性评分。CPI仅使用全局对齐进行粗略不匹配移除，然后通过是否在受控替换下短语显著影响图像-文本评分来对幸存池进行排名。我们将CPI框架为一阶短语敏感性信号，而非接地或识别结果，并在CC3M规模上评估。按此信号排名产生一个50%的数据子集，在VL-CheckList-VG关系任务上比完整数据基线提高+1.91，在匹配预算下比仅对齐过滤提高+1.00，同时提高SugarCrepe整体表现并保持泛化转移。CPI是损失正交的：应用不变于NegCLIP，它进一步在VL-CheckList-VG关系任务上提高+3.84，并在主要文本中获得额外的CE-CLIP收益。
英文摘要：
CLIP-style contrastive pretraining typically curates web-scale image-text pairs using sample-level filtering signals, often based on pair-level alignment. We show that this signal saturates: once coarse mismatches are removed, stricter global filtering no longer tracks the compositional supervision provided by the retained captions. The reason is structural - a global score conflates whether a pair is broadly plausible with whether the individual object, attribute, and relation phrases inside the caption materially support the image-text match. The latter is what compositional generalization demands, yet pair-level filters are blind to it. We address this with Counterfactual Phrase Intervention (CPI), a phrase-level curation framework that converts controlled nonce-token substitutions into image-conditioned phrase-sensitivity scores. CPI uses global alignment only for coarse mismatch removal, then ranks the surviving pool by whether caption phrases measurably affect the image-text score under controlled substitution. We frame CPI as a first-order phrase-sensitivity signal rather than a grounding or identification result, and evaluate it at CC3M scale. Ranking by this signal yields a 50%-data subset that improves VL-CheckList-VG Relation by +1.91 over the full-data baseline and +1.00 over alignment-only filtering at matched budget, while improving SugarCrepe overall and preserving general transfer. CPI is loss-orthogonal: applied unchanged to NegCLIP, it further improves VL-CheckList-VG Relation by +3.84, with additional CE-CLIP gains in the main text.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
