探究在可控反事实扰动下的视觉语义对齐中的各向异性
Investigating Anisotropy in Visual Grounding under Controlled Counterfactual Perturbations
AI总结：
本文研究了在可控反事实扰动下视觉语义对齐中的各向异性影响，通过对比两种不同嵌入几何的模型发现，余弦相似度与近似行为无显著关联，表明各向异性不足以解释反事实错误。
摘要：
视觉语义基准假设引用表达描述的对象始终存在于图像中，因此地缘模型很少在语义不匹配的标题下进行评估。在这些情况下，模型经常表现出近似行为，生成一个合理的边界框，只满足表达的一部分（例如，保留原始对象而忽略修改的上下文线索）。因为不匹配的标题代表了现实中的边缘情况，这种行为影响了可靠性，并从可解释性角度引发了担忧。识别其根本原因对于提高模型的忠实度和可解释性至关重要。采用机制可解释性观点，本文研究嵌入各向异性是否有助于反事实失败。引入一种相似性控制的反事实标题生成协议，系统地扰动预定义嵌入相似度区间内的对象或上下文组件，从而对齐行为作为函数的细粒度分析。在两个具有明显不同嵌入几何的Transformer模型（BERT基于TransVG和CLIP基于SwimVG）上的实验发现，余弦相似度与近似行为无显著关联。这些发现表明，各向异性本身不足以解释反事实错误，鲁棒性需要研究嵌入空间的更细粒度几何属性。
英文摘要：
Visual Grounding benchmarks assume that the object described by a referring expression is always present in the image, and grounding models are therefore rarely evaluated under semantically mismatched captions. In such cases, models frequently exhibit approximation behavior, producing a plausible bounding box that satisfies only part of the expression (\eg, preserving the original object while ignoring modified contextual cues). Because mismatched captions represent realistic edge cases, this behavior compromises reliability and raises concerns from an explainability perspective. Identifying its underlying causes is thus essential for improving model faithfulness and interpretability. Adopting a mechanistic interpretability viewpoint, this work examines whether embedding anisotropy contributes to counterfactual failures. A similarity-controlled counterfactual caption generation protocol is introduced to systematically perturb object or contextual components within predefined embedding similarity intervals, enabling a fine-grained analysis of grounding behavior as a function of alignment. Experiments on two Transformer-based models with markedly different embedding geometries (BERT-based TransVG and CLIP-based SwimVG) reveal no meaningful correlation between cosine similarity and approximation. These findings suggest that anisotropy alone does not account for counterfactual errors, and that robustness requires investigating finer-grained geometric properties of the embedding space.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
