深权空间中的偏局部熵和各向异性
Partial local entropy and anisotropy in deep weight spaces
摘要：
我们改进了最近提出的一类局部熵损失函数，限制平滑正则化，只有一个子集的权重。新的损失函数被称为部分局部熵。它们可以适应权重空间的各向异性，从而优于它们的各向同性对应物。我们支持的理论分析与实验的图像分类任务进行多层，全连接和卷积神经网络。本研究提出了如何更好地利用深景观的各向异性的性质，并提供了直接探测的形状遇到的最小值的随机梯度下降算法。作为一个副产品，我们观察到一个渐近动力学制度在后期的训练时间，所有层的温度服从一个共同的冷却行为。
英文摘要：
We refine a recently-proposed class of local entropic loss functions by restricting the smoothening regularization to only a subset of weights. The new loss functions are referred to as partial local entropies. They can adapt to the weight-space anisotropy, thus outperforming their isotropic counterparts. We support the theoretical analysis with experiments on image classification tasks performed with multi-layer, fully-connected and convolutional neural networks. The present study suggests how to better exploit the anisotropic nature of deep landscapes and provides direct probes of the shape of the minima encountered by stochastic gradient descent algorithms. As a by-product, we observe an asymptotic dynamical regime at late training times where the temperature of all the layers obeys a common cooling behavior.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
