你的CLIP有164个噪声维度：探索对比性预训练视觉-语言变换器的嵌入协方差特征谱
Your CLIP has 164 dimensions of noise: Exploring the embeddings covariance eigenspectrum of contrastively pretrained vision-language transformers
AI总结：
本文通过协方差矩阵谱分解，揭示对比预训练VLMs的潜在空间中多模态噪声结构，发现去除共享噪声维度对下游任务性能无害，提供对现代VLMs表征结构的新见解。
摘要：
对比性预训练的视觉-语言模型（VLMs）是强大的特征提取器。然而，它们共享的潜在空间易出现结构性异常，并成为非语义、多模态噪声的存储库。为解决这一现象，我们采用协方差矩阵的谱分解，将VLM潜在空间分解为多模态语义信号成分和共享噪声子空间。我们观察到这种噪声几何结构在不同数据子集之间表现出强子组不变性。关键的是，修剪这些共享噪声维度主要无害，保留或主动提升下游任务性能。通过将真实语义信号与人工噪声分离，本文为现代VLMs的表征结构提供了新的机理见解，表明其大量潜在几何由共享的架构级噪声而非仅任务相关语义所主导。
英文摘要：
Contrastively pre-trained Vision-Language Models (VLMs) serve as powerful feature extractors. Yet, their shared latent spaces are prone to structural anomalies and act as repositories for non-semantic, multi-modal noise. To address this phenomenon, we employ spectral decomposition of covariance matrices to decompose the VLM latent space into a multi-modal semantic signal component and a shared noise subspace. We observe that this noise geometry exhibits strong subgroup invariance across distinct data subsets. Crucially, pruning these shared noise dimensions is mainly harmless, preserving or actively improving downstream task performance. By isolating true semantic signals from artifactual noise, this work provides new mechanistic insights into the representational structure of modern VLMs, suggesting that a substantial fraction of their latent geometry is governed by shared, architecture-level noise rather than task-relevant semantics alone.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
