It is commonly observed that deep networks trained for classification exhibit class-selective neurons in their early and intermediate layers. Intriguingly, recent studies have shown that these class-selective neurons can be ablated without deteriorating network function. But if class-selective neurons are not necessary, why do they exist? We attempt to answer this question in a series of experiments on ResNet-50s trained on ImageNet. We first show that class-selective neurons emerge during the first few epochs of training, before receding rapidly but not completely; this suggests that class-selective neurons found in trained networks are in fact vestigial remains of early training. With single-neuron ablation experiments, we then show that class-selective neurons are important for network function in this early phase of training. We also observe that the network is close to a linear regime in this early phase; we thus speculate that class-selective neurons appear early in training as quasi-linear shortcut solutions to the classification task. Finally, in causal experiments where we regularize against class selectivity at different points in training, we show that the presence of class-selective neurons early in training is critical to the successful training of the network; in contrast, class-selective neurons can be suppressed later in training with little effect on final accuracy. It remains to be understood by which mechanism the presence of class-selective neurons in the early phase of training contributes to the successful training of networks.
Fast gradient-free activation maximization for neurons in spiking neural networks
Laboratory of Neuronal Intelligence, Institute for Advanced Brain Studies, Lomonosov Moscow State University(神经智能实验室,先进脑科学研究所,洛蒙诺索夫莫斯科国立大学);Artificial Intelligence Research Institute (AIRI)(人工智能研究所);Skolkovo Institute of Science and Technology(斯克罗夫科技研究院);HSE University Moscow, Russia(莫斯科高等经济大学,俄罗斯)
Universal Neurons in GPT-2: Emergence, Persistence, and Functional Impact
University of Waterloo(滑铁卢大学);University of Illinois Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校);Ottawa University(渥太华大学);Algoverse(Algoverse公司);Meta FAIR Lab(Meta FAIR实验室);UCSD(加州大学圣地亚哥分校)
Balanced Excitation and Inhibition are Required for High-Capacity, Noise-Robust Neuronal Selectivity
Columbia University(哥伦比亚大学);Edmond and Lily Safra Center for Brain Sciences(埃德蒙和莉莉·萨弗拉脑科学中心);Racah Institute of Physics(拉卡物理研究所);The Hebrew University of Jerusalem(耶路撒冷希伯来大学);Center for Brain Science(脑科学中心);Harvard University(哈佛大学)