CLIP纹理-形状偏倚的动态演变及其与人类对齐和模型鲁棒性的关系
On the dynamic evolution of CLIP texture-shape bias and its relationship to human alignment and model robustness
AI总结：
本文研究CLIP模型在训练过程中纹理-形状偏倚的演变，揭示其与人类感知对齐及模型鲁棒性的关系。
摘要：
对比语言-图像模型如CLIP已显示出显著的泛化能力。然而，它们的内部视觉表示在训练过程中如何演变，以及这种演变如何与人类感知相关仍不明确。现有分析大多描述完全训练后的模型，使得表示偏倚和感知对齐的动态发展鲜为人知。在本文中，我们对CLIP模型的训练过程进行逐epoch分析，关注纹理-形状偏倚、与人类感知判断的一致性以及对图像噪声的敏感性。通过多个感知基准测试，涵盖低级图像质量评估、中级感知相似性、显著性对应和噪声鲁棒性，我们识别出一种一致的、训练阶段依赖的表示转变。早期训练阶段表现出强纹理偏倚，与低级人类感知测量的对齐性较高，且对高斯噪声扰动的敏感性增加。随着训练的进行，这种纹理偏倚逐渐减弱，取而代之的是更基于形状的表示，同时对噪声的鲁棒性提高，低级感知对齐性下降。重要的是，这些动态在多个CLIP模型规模中一致观察到，表明该现象不特定于特定架构大小。我们的发现提供了关于感知对齐、特征偏倚和鲁棒性在多模态模型训练过程中如何共同演变的实证描述。这项工作揭示了早期低级感知对齐与后期鲁棒性之间的系统性权衡，为视觉-语言模型的表示动态及其与人类视觉处理的关系提供了新的见解。
英文摘要：
Contrastive language-image models such as CLIP have demonstrated remarkable generalization capabilities. However, how their internal visual representations evolve during training and how this evolution relates to human perception remains poorly understood. Most existing analysis characterize fully trained models, leaving the dynamics of representational biases and perceptual alignment largely unexplored. In this work, we present an epoch-by-epoch analysis of CLIP models throughout training, focusing on the evolution of texture-shape bias, alignment with human perceptual judgements, and sensitivity to image noise. Using multiple perceptual benchmarks spanning low-level image quality assessment, mid-level perceptual similarity, saliency correspondence, and noisy robustness, we identify a consistent, training-stage-dependent representational transition. Early training stages exhibit strong texture bias, elevated alignment with low-level human perceptual measures, and increased sensitivity to Gaussian noise perturbations. As training progresses, this texture bias gradually diminishes in favor of more shape-based representations, coinciding with improved robustness to noise and a decline in low-level perceptual alignment. Importantly, these dynamics are consistently observed across multiple CLIP model scales, indicating that the phenomenon is not specific to a particular architecture size. Our findings provide an empirical characterization of how perceptual alignment, feature bias, and robustness co-evolve during multimodal model training. This work reveals a systematic trade-off between early low-level perceptual alignment and later robustness, offering new insights into the representational dynamics of vision-language models and their relationship to human visual processing.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
