arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2305.17409cs.LG
收藏

关于类别选择性神经元在训练早期的特殊作用

On the special role of class-selective neurons in early training

Alchera X;Northwestern University(西北大学);Meta AI (FAIR)(Meta人工智能(FAIR));MosaicML;Aalto University(阿尔托大学)

Omkar Ranadive, Nikhil Thakurdesai, Ari S Morcos, Matthew Leavitt, Stéphane Deny

AI总结:

本文通过ResNet-50在ImageNet上的实验揭示,类别选择性神经元是训练早期出现的准线性捷径解,其早期存在对网络成功训练至关重要,而后期可被抑制且几乎不影响最终准确率。

摘要:

人们普遍观察到,为分类任务训练的深度网络在其早期和中间层中会表现出类别选择性神经元。有趣的是,近期研究表明,这些类别选择性神经元可以在不损害网络功能的情况下被消融。但如果类别选择性神经元并非必要,它们为何存在?我们尝试通过对在ImageNet上训练的ResNet-50进行一系列实验来回答这一问题。我们首先表明,类别选择性神经元在训练的最初几个epoch内出现,随后迅速但未完全消退;这表明在已训练网络中发现的类别选择性神经元实际上是早期训练的遗迹残留。通过单神经元消融实验,我们随后表明,类别选择性神经元在训练的这一早期阶段对网络功能很重要。我们还观察到,网络在这一早期阶段接近线性状态;因此我们推测,类别选择性神经元在训练早期出现,是作为分类任务的准线性捷径解。最后,在训练的不同时间点对类别选择性进行正则化抑制的因果实验中,我们表明,训练早期类别选择性神经元的存在对网络的成功训练至关重要;相比之下,在训练后期抑制类别选择性神经元对最终准确率几乎没有影响。类别选择性神经元在训练早期的存在通过何种机制促进网络的成功训练,仍有待理解。

英文摘要:

It is commonly observed that deep networks trained for classification exhibit class-selective neurons in their early and intermediate layers. Intriguingly, recent studies have shown that these class-selective neurons can be ablated without deteriorating network function. But if class-selective neurons are not necessary, why do they exist? We attempt to answer this question in a series of experiments on ResNet-50s trained on ImageNet. We first show that class-selective neurons emerge during the first few epochs of training, before receding rapidly but not completely; this suggests that class-selective neurons found in trained networks are in fact vestigial remains of early training. With single-neuron ablation experiments, we then show that class-selective neurons are important for network function in this early phase of training. We also observe that the network is close to a linear regime in this early phase; we thus speculate that class-selective neurons appear early in training as quasi-linear shortcut solutions to the classification task. Finally, in causal experiments where we regularize against class selectivity at different points in training, we show that the presence of class-selective neurons early in training is critical to the successful training of the network; in contrast, class-selective neurons can be suppressed later in training with little effect on final accuracy. It remains to be understood by which mechanism the presence of class-selective neurons in the early phase of training contributes to the successful training of networks.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑