arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 1805.07249cs.LGcs.AIcs.NEstat.ML
收藏

基于互信息的动态学习率

Dynamic learning rate using Mutual Information

IBM Research - India(IBM印度研究院)

Shrihari Vasudevan

摘要:

本文演示了使用互信息(MI)进行深度神经网络训练的动态超参数设置。本文研究的具体超参数是学习率。利用输出层与真实结果之间的互信息,通过训练周期动态地设置网络的学习率,并将该思想扩展到逐层设置学习率。证明了两种方法-跟踪互信息的相对变化,另外跟踪其相对于参考测量的值。本文并不试图推荐一个具体的学习率政策。实验表明,互信息可以有效地用于动态设置学习率,并在竞争更好的时间内实现竞争更好的结果。

英文摘要:

This paper demonstrates dynamic hyper-parameter setting, for deep neural network training, using Mutual Information (MI). The specific hyper-parameter studied in this paper is the learning rate. MI between the output layer and true outcomes is used to dynamically set the learning rate of the network through the training cycle; the idea is also extended to layer-wise setting of learning rate. Two approaches are demonstrated - tracking relative change in mutual information and, additionally tracking its value relative to a reference measure. The paper does not attempt to recommend a specific learning rate policy. Experiments demonstrate that mutual information may be effectively used to dynamically set learning rate and achieve competitive to better outcomes in competitive to better time.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑