arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2109.14925cs.LG
收藏

用于超参数优化的谱系群体训练

Genealogical Population-Based Training for Hyperparameter Optimization

ENS, PSL Research University(巴黎高等师范学院,巴黎文理研究大学)

Antoine Scardigli, Paul Fournier, Matteo Vilucchio, David Naccache

AI总结:

提出谱系群体训练(GPBT)方法,利用谱系相关模型的共享历史高效耦合超参数与模型,在计算机视觉任务中降低2-3倍计算成本、提升1%准确率并显著降低方差,且与搜索算法无关。

摘要:

超参数优化(HPO)旨在以最快、最高效的方式寻找学习模型(如神经网络)的最佳超参数(HPs)。最近的HPO算法试图在不考虑获得这些超参数的模型的情况下进行优化,假设对于不同的模型,相同的超参数会产生非常相似的结果。我们摆脱了这一范式,提出了一种对现有方法的新应用,称为谱系群体训练(GPBT)。GPBT通过“谱系”相关模型的共享历史,高效地利用了超参数与模型之间的耦合关系。我们通过实验证明,与当前算法相比,我们的方法将所需的计算成本降低了2到3倍,通常在计算机视觉任务上实现了1%的准确率提升,并将结果的方差降低了一个数量级。我们的方法与搜索算法无关,因此内部搜索例程可以是任何搜索算法,如TPE、GP、CMA或随机搜索。

英文摘要:

HyperParameter Optimization (HPO) aims at finding the best HyperParameters (HPs) of learning models, such as neural networks, in the fastest and most efficient way possible. Most recent HPO algorithms try to optimize HPs regardless of the model that obtained them, assuming that for different models, same HPs will produce very similar results. We break free from this paradigm and propose a new take on preexisting methods that we called Genealogical Population Based Training (GPBT). GPBT, via the shared histories of "genealogically"-related models, exploit the coupling of HPs and models in an efficient way. We experimentally demonstrate that our method cuts down by 2 to 3 times the computational cost required, generally allows a 1% accuracy improvement on computer vision tasks, and reduces the variance of the results by an order of magnitude, compared to the current algorithms. Our method is search-algorithm agnostic so that the inner search routine can be any search algorithm like TPE, GP, CMA or random search.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑