arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2409.11995cs.LG
收藏

解析 Hessian 矩阵:损失函数景观中平滑收敛的关键

Unraveling the Hessian: A Key to Smooth Convergence in Loss Function Landscapes

MIPT(莫斯科物理技术学院);Sber AI

Nikita Kiselev, Andrey Grabovoy

AI总结:

本文研究神经网络损失景观随样本量增加的收敛性,通过理论推导损失值差异上界并在图像分类数据集上实证验证,为理解损失景观几何和样本量确定提供依据。

摘要:

神经网络的损失景观是其训练的一个关键方面,理解其性质对于提升性能至关重要。本文研究了当样本量增加时损失曲面的变化,这是一个此前未被探索的问题。我们从理论上分析了全连接神经网络中损失景观的收敛性,并推导出当向样本中添加新对象时损失函数值差异的上界。我们的实证研究在各种数据集上证实了这些结果,展示了图像分类任务中损失函数曲面的收敛性。我们的发现为神经损失景观的局部几何结构提供了见解,并对样本量确定技术的发展具有启示意义。

英文摘要:

The loss landscape of neural networks is a critical aspect of their training, and understanding its properties is essential for improving their performance. In this paper, we investigate how the loss surface changes when the sample size increases, a previously unexplored issue. We theoretically analyze the convergence of the loss landscape in a fully connected neural network and derive upper bounds for the difference in loss function values when adding a new object to the sample. Our empirical study confirms these results on various datasets, demonstrating the convergence of the loss function surface for image classification tasks. Our findings provide insights into the local geometry of neural loss landscapes and have implications for the development of sample size determination techniques.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑