从 Token 级因果视角理解视觉-语言组合性的困难
Understanding Hardness of Vision-Language Compositionality from A Token-level Causal Lens
AI总结：
本文提出 token 感知因果表示学习框架，通过 token 级结构因果模型证明 CLIP 的组合脆弱性源于组合不可识别性，揭示了伪最优编码器对原子概念操作不敏感导致困难负样本失败的机制。
摘要：
对比语言-图像预训练（CLIP）通过在共享嵌入空间中对齐图像和文本，提供了强大的跨模态泛化能力，但它在对象、属性和关系的组合推理上持续失败，通常表现得像一个词袋匹配器。先前的因果解释通常将文本建模为单个向量，掩盖了 token 级结构，导致核心现象（如提示敏感性和在困难负样本上的失败）无法得到解释。我们通过一个基于序列化语言 token 结构因果模型（SCM）的 token 感知因果表示学习（CRL）框架来填补这一空白。我们的理论将块可识别性扩展至 token 化文本，证明了 CLIP 的对比目标在句子级和 token 级 SCM 下均能恢复模态不变潜变量。关键的是，token 粒度为 CLIP 的组合脆弱性提供了首个原理性解释：组合不可识别性。我们证明了存在伪最优文本编码器，它们能实现完美的模态不变对齐，但在数学上对原子概念的 SWAP、REPLACE 和 ADD 操作不敏感，因此尽管优化了与真最优编码器相同的训练目标，却无法区分正确描述与困难负样本。分析进一步通过模态差距将语言侧的不可识别性与视觉侧的失败联系起来，并展示了迭代组合算子如何加剧困难程度，从而为改进负样本挖掘策略提供了动机。
英文摘要：
Contrastive Language-Image Pre-training (CLIP) delivers strong cross modal generalization by aligning images and texts in a shared embedding space, yet it persistently fails at compositional reasoning over objects, attributes, and relations often behaving like a bag-of-words matcher. Prior causal accounts typically model text as a single vector, obscuring token-level structure and leaving core phenomena-such as prompt sensitivity and failures on hard negatives unexplained. We address this gap with a token-aware causal representation learning (CRL) framework grounded in a sequential, language-token SCM. Our theory extends block identifiability to tokenized text, proving that CLIP's contrastive objective can recover the modal-invariant latent variable under both sentence-level and token-level SCMs. Crucially, token granularity yields the first principled explanation of CLIP's compositional brittleness: composition nonidentifiability. We show the existence of pseudo-optimal text encoders that achieve perfect modal-invariant alignment yet are provably insensitive to SWAP, REPLACE, and ADD operations over atomic concepts, thereby failing to distinguish correct captions from hard negatives despite optimizing the same training objective as true-optimal encoders. The analysis further links language-side nonidentifiability to visual-side failures via the modality gap and shows how iterated composition operators compound hardness, motivating improved negative mining strategies.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
