arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2301.13247cs.LGcs.AI
收藏

元学习自适应损失函数

Meta-Learning Adaptive Loss Functions

Victoria University of Wellington(惠灵顿维多利亚大学)

Christian Raymond, Qi Chen, Bing Xue, Mengjie Zhang

AI总结:

针对现有离线元学习损失函数因仅考虑训练初期而导致最终性能不佳的问题,提出在线自适应更新损失函数的新方法,在多种架构和数据集上均优于交叉熵和离线学习技术。

摘要:

损失函数学习是一种新的元学习范式,旨在自动化为机器学习模型设计损失函数这一关键任务。现有的损失函数学习技术已展现出令人鼓舞的结果,通常能改善模型的训练动态和最终推理性能。然而,这些技术的一个显著局限是损失函数以离线方式进行元学习,其元目标仅考虑训练的最开始几步,这比通常用于训练深度神经网络的时间跨度要短得多。这导致显著偏向于在训练初期表现良好但在训练结束时表现不佳的损失函数。为解决此问题,我们提出了一种新的损失函数学习技术,在对基础模型参数进行每次更新后,在线自适应地更新损失函数。实验结果表明,我们提出的方法在多种神经网络架构和数据集上始终优于交叉熵损失和离线损失函数学习技术。

英文摘要:

Loss function learning is a new meta-learning paradigm that aims to automate the essential task of designing a loss function for a machine learning model. Existing techniques for loss function learning have shown promising results, often improving a model's training dynamics and final inference performance. However, a significant limitation of these techniques is that the loss functions are meta-learned in an offline fashion, where the meta-objective only considers the very first few steps of training, which is a significantly shorter time horizon than the one typically used for training deep neural networks. This causes significant bias towards loss functions that perform well at the very start of training but perform poorly at the end of training. To address this issue we propose a new loss function learning technique for adaptively updating the loss function online after each update to the base model parameters. The experimental results show that our proposed method consistently outperforms the cross-entropy loss and offline loss function learning techniques on a diverse range of neural network architectures and datasets.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑