Items or Relations -- what do Artificial Neural Networks learn?
Institute of Neuroinformatics, University of Zurich and ETH Zurich（苏黎世大学与苏黎世联邦理工学院神经信息学研究所）；Neuroscience Center Zurich, University of Zurich and ETH Zurich（苏黎世大学与苏黎世联邦理工学院苏黎世神经科学中心）
What has an Artificial Neural Network (ANN) learned after being successfully trained to solve a task - the set of training items or the relations between them? This question is difficult to answer for modern applied ANNs because of their enormous size and complexity. Therefore, here we consider a low-dimensional network and a simple task, i.e., the network has to reproduce a set of training items identically. We construct the family of solutions analytically and use standard learning algorithms to obtain numerical solutions. These numerical solutions differ depending on the optimization algorithm and the weight initialization and are shown to be particular members of the family of analytical solutions. In this simple setting, we observe that the general structure of the network weights represents the training set's symmetry group, i.e., the relations between training items. As a consequence, linear networks generalize, i.e., reproduce items that were not part of the training set but are consistent with the symmetry of the training set. In contrast, non-linear networks tend to learn individual training items and show associative memory. At the same time, their ability to generalize is limited. A higher degree of generalization is obtained for networks whose activation function contains a linear regime, such as tanh. Our results suggest ANN's ability to generalize - instead of learning items - could be improved by generating a sufficiently big set of elementary operations to represent relations and strongly depends on the applied non-linearity.
The Why, What and How of Artificial General Intelligence Chip Development
Kerala University of Digital Sciences, Innovation and Technology (KUDSIT)（喀拉拉邦数字科学、创新与技术大学）；Indian Institute of Information Technology and Management, Kerala（喀拉拉邦印度信息技术与管理学院）
