半真信息破坏基于相似性的检索
Half-Truths Break Similarity-Based Retrieval
AI总结：
CS-CLIP通过分解标题为实体和关系单元并改进监督方法，提升了基于相似性检索的准确性。
摘要：
当文本描述增加额外细节时，图像-文本相似性应下降，如果该细节是错误的。我们证明CLIP风格的双编码器经常违反这一直觉：在原本正确的描述中添加一个合理但错误的对象或关系，可以增加相似性分数。我们称这种情况为半真信息。在COCO数据集上，CLIP只在40.6%的情况下偏好正确的简短描述，当添加的细节是关系时，性能降至32.9%。我们追溯这种弱点到对标题部分的弱监督：对比训练对完整句子进行对齐，但并未明确要求个体实体和关系被地面化。我们提出了CS-CLIP（组件监督CLIP），它将标题分解为实体和关系单元，为每个单元构建最小编辑的对照，并通过微调模型来评分正确的单元高于其对照，同时保持标准双编码器推断。CS-CLIP将半真信息准确性提升到69.3%，并在已建立的组合性基准上平均提高了5.7个点，表明减少半真信息错误与更广泛的整体组合理解提升相关。代码可在https://github.com/kargibora/CS-CLIP上公开获取。
英文摘要：
When a text description is extended with an additional detail, image-text similarity should drop if that detail is wrong. We show that CLIP-style dual encoders often violate this intuition: appending a plausible but incorrect object or relation to an otherwise correct description can increase the similarity score. We call such cases half-truths. On COCO, CLIP prefers the correct shorter description only 40.6% of the time, and performance drops to 32.9% when the added detail is a relation. We trace this vulnerability to weak supervision on caption parts: contrastive training aligns full sentences but does not explicitly enforce that individual entities and relations are grounded. We propose CS-CLIP (Component-Supervised CLIP), which decomposes captions into entity and relation units, constructs a minimally edited foil for each unit, and fine-tunes the model to score the correct unit above its foil while preserving standard dual-encoder inference. CS-CLIP raises half-truth accuracy to 69.3% and improves average performance on established compositional benchmarks by 5.7 points, suggesting that reducing half-truth errors aligns with broader gains in compositional understanding. Code is publicly available at: https://github.com/kargibora/CS-CLIP
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
