归纳瓶颈：视觉Transformer中表征稀疏性的数据驱动出现
The Inductive Bottleneck: Data-Driven Emergence of Representational Sparsity in Vision Transformers
AI总结：
研究视觉Transformer中“归纳瓶颈”现象，通过分析不同数据集上DINO训练的ViT逐层有效编码维度，发现瓶颈深度与任务语义抽象相关，纹理丰富数据集保持高秩表示，以对象为中心的数据集促使网络学习瓶颈隔离语义特征。
摘要：
视觉Transformer（ViT）缺乏卷积神经网络（CNN）固有的分层归纳偏差，理论上允许它们在所有层中保持高维表示。然而，最近的观察表明，ViT经常自发地呈现出“U形”熵分布，即在中间层压缩信息，然后在最终分类前扩展信息。在这项工作中，我们证明了这种“归纳瓶颈”不是架构产物，而是数据依赖的适应。通过分析在不同组成复杂性的数据集（加州大学默塞德分校、微小图像网和CIFAR-100）上用DINO训练的ViT的逐层有效编码维度（EED），我们表明瓶颈的深度与任务所需的语义抽象密切相关。我们发现，虽然纹理丰富的数据集始终保留高秩表示，但以对象为中心的数据集会促使网络抑制中间层的高频信息，有效地“学习”一个瓶颈来隔离语义特征。
英文摘要：
Vision Transformers (ViTs) lack the hierarchical inductive biases inherent to Convolutional Neural Networks (CNNs), theoretically allowing them to maintain high-dimensional representations throughout all layers. However, recent observations suggest ViTs often spontaneously manifest a "U-shaped" entropy profile-compressing information in middle layers before expanding it for the final classification. In this work, we demonstrate that this "Inductive Bottleneck" is not an architectural artifact, but a data-dependent adaptation. By analyzing the layer-wise Effective Encoding Dimension (EED) of DINO-trained ViTs across datasets of varying compositional complexity (UC Merced, Tiny ImageNet, and CIFAR-100), we show that the depth of the bottleneck correlates strongly with the semantic abstraction required by the task. We find that while texture-heavy datasets preserve high-rank representations throughout, object-centric datasets drive the network to dampen high-frequency information in middle layers, effectively "learning" a bottleneck to isolate semantic features.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
