基于循环学习率的多保真度超参数优化
Improving Multi-fidelity Optimization with a Recurring Learning Rate for Hyperparameter Tuning
摘要：
尽管卷积神经网络（CNN）不断发展，但它们的性能令人惊讶地依赖于超参数的选择。然而，由于现代CNN的训练时间很长，因此有效地探索大型超参数搜索空间仍然具有挑战性。多保真度优化使得能够通过提前终止不希望的配置来探索更多的超参数配置。然而，这通常导致选择次优配置，因为使用高性能配置的训练通常在早期阶段收敛缓慢。在本文中，我们提出了具有循环学习率的多保真度优化（MORL），它将CNN的优化过程融入多保真度优化中。MORL消除了慢启动器的问题，并实现了更精确的低保真度近似。我们在一般图像分类、迁移学习和半监督学习上的综合实验证明了MORL相对于其他多保真度优化方法（如连续减半算法（SHA）和Hyperband）的有效性。此外，它实现了显着的性能改进，手动调整超参数配置在一个实际的预算。
英文摘要：
Despite the evolution of Convolutional Neural Networks (CNNs), their performance is surprisingly dependent on the choice of hyperparameters. However, it remains challenging to efficiently explore large hyperparameter search space due to the long training times of modern CNNs. Multi-fidelity optimization enables the exploration of more hyperparameter configurations given budget by early termination of unpromising configurations. However, it often results in selecting a sub-optimal configuration as training with the high-performing configuration typically converges slowly in an early phase. In this paper, we propose Multi-fidelity Optimization with a Recurring Learning rate (MORL) which incorporates CNNs' optimization process into multi-fidelity optimization. MORL alleviates the problem of slow-starter and achieves a more precise low-fidelity approximation. Our comprehensive experiments on general image classification, transfer learning, and semi-supervised learning demonstrate the effectiveness of MORL over other multi-fidelity optimization methods such as Successive Halving Algorithm (SHA) and Hyperband. Furthermore, it achieves significant performance improvements over hand-tuned hyperparameter configuration within a practical budget.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
