揭示神经网络中学习子空间的已利用秩
Revealing the Utilized Rank of Subspaces of Learning in Neural Networks
AI总结：
本文提出一种数据驱动变换，将神经网络权重投影到数据与权重交互的子空间，揭示其低秩结构，发现多数模型仅利用部分可用空间，并显著减少参数且几乎不损失准确率。
摘要：
在这项工作中，我们研究神经网络学习到的权重在多大程度上利用了可供其使用的空间。这一概念与容量相关，但还额外包含了网络架构与数据集之间的交互。大多数学习到的权重看起来是满秩的，因此不适合进行低秩分解。这具有欺骗性地暗示权重正在利用可供其使用的全部空间。我们提出了一种简单的数据驱动变换，将权重投影到数据与权重发生交互的子空间上。这保留了层的函数映射，并揭示了其低秩结构。在我们的研究结果中，我们得出结论：大多数模型只利用了可用空间的一小部分。例如，对于在ImageNet上训练的ViTB-16和ViTL-16，平均层利用率分别为35%和20%。我们的变换使参数分别减少到50%和25%，同时在微调后导致的准确率下降不到0.2%。我们还表明，自监督预训练可将这一利用率提高到70%，这证明了其适用于下游任务。
英文摘要：
In this work, we study how well the learned weights of a neural network utilize the space available to them. This notion is related to capacity, but additionally incorporates the interaction of the network architecture with the dataset. Most learned weights appear to be full rank, and are therefore not amenable to low rank decomposition. This deceptively implies that the weights are utilizing the entire space available to them. We propose a simple data-driven transformation that projects the weights onto the subspace where the data and the weight interact. This preserves the functional mapping of the layer and reveals its low rank structure. In our findings, we conclude that most models utilize a fraction of the available space. For instance, for ViTB-16 and ViTL-16 trained on ImageNet, the mean layer utilization is 35% and 20% respectively. Our transformation results in reducing the parameters to 50% and 25% respectively, while resulting in less than 0.2% accuracy drop after fine-tuning. We also show that self-supervised pre-training drives this utilization up to 70%, justifying its suitability for downstream tasks.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
