arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2006.10424cs.LGcs.AIstat.ML
收藏

监控神经网络训练进度的权空间研究

An Investigation of the Weight Space to Monitor the Training Progress of Neural Networks

University of St.Gallen(圣加仑大学)

Konstantin Schürholt, Damian Borth

摘要:

安全使用深度神经网络(DNN)需要仔细测试。然而,部署的模型通常会被进一步训练以提高性能。由于严格的测试和评估是昂贵的,触发器需要确定模型的变化程度。在本文中,我们研究了DNN模型的结构权重空间,可以利用这一点。我们的研究结果表明,DNN模型在权重空间中沿着独特、平滑的轨迹演化,可用于跟踪DNN的训练进度。我们假设轨迹的曲率和平滑度以及沿着它的步长可能包含关于训练状态以及潜在域偏移的信息。我们证明了模型轨迹可以分离,轨迹上的检查点顺序可以恢复,这可以作为DNN模型版本化的第一步。

英文摘要:

Safe use of Deep Neural Networks (DNNs) requires careful testing. However, deployed models are often trained further to improve in performance. As rigorous testing and evaluation is expensive, triggers are in need to determine the degree of change of a model. In this paper we investigate the weight space of DNN models for structure that can be exploited to that end. Our results show that DNN models evolve on unique, smooth trajectories in weight space which can be used to track DNN training progress. We hypothesize that curvature and smoothness of the trajectories as well as step length along it may contain information on the state of training as well as potential domain shifts. We show that the model trajectories can be separated and the order of checkpoints on the trajectories recovered, which may serve as a first step towards DNN model versioning.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑