arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2212.04183cs.LG
收藏

注意差距:衡量跨多个目标的泛化性能

Mind the Gap: Measuring Generalization Performance Across Multiple Objectives

Albert-Ludwigs-Universität Freiburg(阿尔伯特-路德维希-弗莱堡大学);Ludwig-Maximilians-Universität München(慕尼黑大学);Bosch Center for Artificial Intelligence(博世人工智能中心)

Matthias Feurer, Katharina Eggensperger, Edward Bergman, Florian Pfisterer, Bernd Bischl, Frank Hutter

AI总结:

针对多目标超参数优化方法在验证集到测试集迁移时部分模型失去 Pareto 最优性导致性能难以量化的问题,本文提出一种新的评估协议,用于衡量 MHPO 方法的泛化性能并支持优化实验间的比较。

摘要:

现代机器学习模型在构建时通常会考虑多个目标,例如在最小化推理时间的同时最大化准确率。多目标超参数优化(MHPO)算法会返回此类候选模型,而 Pareto 前沿的近似被用于评估它们的性能。在实践中,我们还希望衡量从验证集转移到测试集时的泛化能力。然而,部分模型可能不再是 Pareto 最优的,这使得在测试集上评估时,如何量化 MHPO 方法的性能变得不明确。为解决这一问题,我们提出了一种新颖的评估协议,该协议能够衡量 MHPO 方法的泛化性能,并研究其在比较两个优化实验方面的能力。

英文摘要:

Modern machine learning models are often constructed taking into account multiple objectives, e.g., minimizing inference time while also maximizing accuracy. Multi-objective hyperparameter optimization (MHPO) algorithms return such candidate models, and the approximation of the Pareto front is used to assess their performance. In practice, we also want to measure generalization when moving from the validation to the test set. However, some of the models might no longer be Pareto-optimal which makes it unclear how to quantify the performance of the MHPO method when evaluated on the test set. To resolve this, we provide a novel evaluation protocol that allows measuring the generalization performance of MHPO methods and studying its capabilities for comparing two optimization experiments.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑