超越压缩：量化视觉表示中的频谱可访问性
Beyond Compression: Quantifying Spectral Accessibility in Vision Representations
AI总结：
通过残差频谱损失（RSL）测量线性可恢复的带限傅里叶能量，研究视觉语言模型中投影层对表示频谱结构的影响，发现CLIP和DINOv2中频谱可访问性随深度非单调变化，中间层峰值后下降，且CLIP的投影是频谱中性的，而DINOv2的[CLS]池化导致频谱结构损失。
摘要：
视觉语言模型通过学习的投影层将视觉特征映射到共享嵌入空间，但目前尚不清楚这些变换如何改变视觉信息的结构。本研究通过空间频率可访问性（以从模型表示中线性恢复带限傅里叶能量的能力衡量）来考察表示的变化。为隔离降维之外的影响，我们引入了残差频谱损失（RSL），该损失相对于维度匹配的随机投影基线评估变化。为减少优化带来的混杂效应，分析使用所有参数冻结的预训练模型。实验结果显示，在ImageNet和MS-COCO数据集上，CLIP和DINOv2中可访问性随频率一致变化。频谱可访问性随深度呈非单调轨迹，在中间层达到峰值，然后向输出表示下降。最终变换因架构而异：CLIP的学习投影是频谱中性的，变化可由压缩解释，而DINOv2的[CLS]池化导致整个频谱的结构性损失。这些发现表明中间层和池化机制是现代视觉编码器中频谱变换的主要驱动因素。
英文摘要：
Vision-language models map visual features into a shared embedding space through learned projection layers, yet it remains unclear how these transformations alter the structure of visual information. This study examines changes in representation through spatial-frequency accessibility, measured by the linear recoverability of band-limited Fourier energy from model representations. To isolate effects beyond dimensionality reduction, we introduce Residual Spectral Loss (RSL), which evaluates changes relative to a dimension-matched random projection baseline. To reduce confounding effects from optimization, the analysis uses pretrained models with all parameters frozen. The experimental results show consistent frequency-dependent changes in accessibility across CLIP and DINOv2 on ImageNet and MS-COCO datasets. Spectral accessibility follows a non-monotonic trajectory across depth, peaking at intermediate layers before decreasing toward the output representation. The final transformation differs across architectures: CLIP's learned projection is spectrally neutral, with changes explained by compression, whereas DINOv2's [CLS] pooling induces a structured loss across the spectrum. These findings identify intermediate layers and pooling mechanisms as primary drivers of spectral transformation in modern vision encoders.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
