AutoLoss:学习离散时间表的交替优化
AutoLoss: Learning Discrete Schedules for Alternate Optimization
摘要:
许多机器学习问题涉及相对于不同的参数集迭代地和交替地优化不同的任务目标。适当地调度任务目标或一组参数的优化通常对收敛的质量至关重要。在本文中,我们提出了AutoLoss,一个自动学习和确定优化时间表的元学习框架。AutoLoss提供了一种通用的方法来表示和学习元数据中的离散优化计划,允许在ML问题中进行动态和数据驱动的计划,这些问题涉及不同参数或不同损失目标的交替更新。我们将AutoLoss应用于四个ML任务:d元二次回归,使用多层感知器(MLP)进行分类,使用GANs进行图像生成,以及多任务神经机器翻译(NMT)。我们表明,AutoLoss控制器能够捕获更好的优化时间表的分布,从而在所有四个任务上实现更高的收敛质量。经过训练的AutoLoss控制器是可推广的-它可以指导和改进具有不同规格或不同数据集的新任务模型的学习。
英文摘要:
Many machine learning problems involve iteratively and alternately optimizing different task objectives with respect to different sets of parameters. Appropriately scheduling the optimization of a task objective or a set of parameters is usually crucial to the quality of convergence. In this paper, we present AutoLoss, a meta-learning framework that automatically learns and determines the optimization schedule. AutoLoss provides a generic way to represent and learn the discrete optimization schedule from metadata, allows for a dynamic and data-driven schedule in ML problems that involve alternating updates of different parameters or from different loss objectives. We apply AutoLoss on four ML tasks: d-ary quadratic regression, classification using a multi-layer perceptron (MLP), image generation using GANs, and multi-task neural machine translation (NMT). We show that the AutoLoss controller is able to capture the distribution of better optimization schedules that result in higher quality of convergence on all four tasks. The trained AutoLoss controller is generalizable -- it can guide and improve the learning of a new task model with different specifications, or on different datasets.
机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。