SGD与自适应学习规则学到的表示：改变神经网络稀疏性与选择性的条件
Representations learnt by SGD and Adaptive learning rules: Conditions that vary sparsity and selectivity in neural networks
AI总结：
本文研究不同优化器、学习率和批量大小等条件下神经网络中稀疏性与选择性自然出现的规律，并在MNIST任务上通过实证分析揭示这些条件与表示性质及测试准确率之间的关系。
摘要：
从人类大脑的角度来看，持续学习能够在不产生相互干扰的情况下执行多种任务。减少相互干扰的一种有效方式可以在神经元的稀疏性和选择性中找到。根据Aljundi等人和Hadsell等人的研究，在表示层面施加稀疏性对持续学习是有利的，因为稀疏的神经元激活促使参数之间的重叠更少，从而减少干扰。类似地，高度选择性的神经网络也可能引起更少的干扰，因为神经元中特定的响应会降低与其他参数重叠的机会。考虑到人类大脑在整个生命周期中进行持续学习，寻找稀疏性和选择性自然出现的条件可能为理解大脑如何运作提供洞见。本文研究了在神经网络中自然增加稀疏性和选择性的各种条件。本文在MNIST分类任务中使用Hoyer稀疏性度量和CCMAS选择性度量测试了不同的优化器。必须指出的是，关于各种条件下稀疏性和选择性自然发生的研究，迄今为止在神经科学或机器学习的任何领域都尚未得到承认。本文发现，特定条件会增加稀疏性和选择性，例如应用较大的学习率和降低批量大小。除了条件、稀疏性和选择性之间的关系之外，以下内容将基于实证分析进行讨论：1. 稀疏性与选择性之间的关系；2. 测试准确率、稀疏性和选择性之间的关系。
英文摘要：
From the point of view of the human brain, continual learning can perform various tasks without mutual interference. An effective way to reduce mutual interference can be found in sparsity and selectivity of neurons. According to Aljundi et al. and Hadsell et al., imposing sparsity at the representational level is advantageous for continual learning because sparse neuronal activations encourage less overlap between parameters, resulting in less interference. Similarly, highly selective neural networks are likely to induce less interference since particular response in neurons will reduce the chance of overlap with other parameters. Considering that the human brain performs continual learning over the lifespan, finding conditions where sparsity and selectivity naturally arises may provide insight for understanding how the brain functions. This paper investigates various conditions that naturally increase sparsity and selectivity in a neural network. This paper tested different optimizers with Hoyer's sparsity metric and CCMAS selectivity metric in MNIST classification task. It is essential to note that investigations on the natural occurrence of sparsity and selectivity concerning various conditions have not been acknowledged in any sector of neuroscience nor machine learning until this day. This paper found that particular conditions increase sparsity and selectivity such as applying a large learning rate and lowering a batch size. In addition to the relationship between the condition, sparsity, and selectivity, the following will be discussed based on empirical analysis: 1. The relationship between sparsity and selectivity and 2. The relationship between test accuracy, sparsity, and selectivity.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
