arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2006.04298cs.LGcs.CVstat.ML
收藏

多步估计在基于先验知识的元学习中的应用

Multi-step Estimation for Gradient-based Meta-learning

SK Telecom(SK电讯)

Jin-Hwa Kim, Junyoung Park, Yongseok Choi

摘要:

基于元学习的元学习方法已经在少样本学习、迁移学习和广泛的其他领域中取得了成功。尽管它的效率和简单性,计算海森矩阵与大内存占用的负担是在大规模应用中的关键挑战。为了解决这个问题,我们提出了一种简单而直接的方法,通过在内部步骤的窗口中重用相同的梯度来降低成本。我们描述了拉格朗日形式主义的多步估计的动力学,并讨论了如何减少评估二阶导数估计的动力学。为了验证我们的方法,我们在Meta迁移学习和多个设置的少样本学习任务上进行了实验。元迁移的实验强调了训练元网络的适用性,而其他近似方法是有限的。对于少样本学习,我们评估了与流行基线相比的时间和内存复杂度。我们表明,我们的方法显着减少了训练时间和内存使用,保持了竞争力的准确性,甚至在某些情况下表现出色。

英文摘要:

Gradient-based meta-learning approaches have been successful in few-shot learning, transfer learning, and a wide range of other domains. Despite its efficacy and simplicity, the burden of calculating the Hessian matrix with large memory footprints is the critical challenge in large-scale applications. To tackle this issue, we propose a simple yet straightforward method to reduce the cost by reusing the same gradient in a window of inner steps. We describe the dynamics of the multi-step estimation in the Lagrangian formalism and discuss how to reduce evaluating second-order derivatives estimating the dynamics. To validate our method, we experiment on meta-transfer learning and few-shot learning tasks for multiple settings. The experiment on meta-transfer emphasizes the applicability of training meta-networks, where other approximations are limited. For few-shot learning, we evaluate time and memory complexities compared with popular baselines. We show that our method significantly reduces training time and memory usage, maintaining competitive accuracies, or even outperforming in some cases.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑