负学习率的元学习
Meta-learning with negative learning rates
摘要:
深度学习模型需要大量数据才能表现良好。当目标任务的数据稀缺时,我们可以转移通过类似任务训练获得的知识,以快速学习目标。一种成功的方法是元学习,或“学习学习”任务的分布,其中“学习”由外循环表示,而“学习”由梯度下降的内循环表示。然而,最近的一些实证研究认为,内部循环是不必要的,更简单的模型同样有效,甚至更好。我们研究了MAML的性能作为内环的学习率的函数,其中零学习率意味着没有内环。利用随机矩阵理论和线性模型的精确解,给出了MAML在混合线性回归和过参数非线性回归中检验损失的代数表达式.令人惊讶的是,虽然适应的最佳学习率是正的,但我们发现训练的最佳学习率总是负的,这是以前从未考虑过的设置。因此,不仅性能通过将学习率降低到零来增加,如最近的工作所建议的,而且性能还可以通过将学习率降低到负值来进一步增加。这些结果有助于澄清在什么情况下元学习表现最好。
英文摘要:
Deep learning models require a large amount of data to perform well. When data is scarce for a target task, we can transfer the knowledge gained by training on similar tasks to quickly learn the target. A successful approach is meta-learning, or "learning to learn" a distribution of tasks, where "learning" is represented by an outer loop, and "to learn" by an inner loop of gradient descent. However, a number of recent empirical studies argue that the inner loop is unnecessary and more simple models work equally well or even better. We study the performance of MAML as a function of the learning rate of the inner loop, where zero learning rate implies that there is no inner loop. Using random matrix theory and exact solutions of linear models, we calculate an algebraic expression for the test loss of MAML applied to mixed linear regression and nonlinear regression with overparameterized models. Surprisingly, while the optimal learning rate for adaptation is positive, we find that the optimal learning rate for training is always negative, a setting that has never been considered before. Therefore, not only does the performance increase by decreasing the learning rate to zero, as suggested by recent work, but it can be increased even further by decreasing the learning rate to negative values. These results help clarify under what circumstances meta-learning performs best.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。