用于HPO基准测试事后分析的线性混合效应模型
LMEMs for post-hoc analysis of HPO Benchmarking
AI总结：
本文提出用线性混合效应模型对HPO基准测试进行事后显著性检验，以揭示平均性能掩盖的方法间差异，并通过PriorBand数据案例验证其能发现新见解。
摘要：
机器学习（ML）和深度学习（DL）中调参超参数的重要性已通过实证研究和应用得到确立，这从社区不断增加的新超参数优化（HPO）算法和基准中可见一斑。然而，当前使用多个数据集上平均性能的基准测试实践可能掩盖HPO方法之间的关键差异，尤其是在成对比较中。在本工作中，我们应用基于线性混合效应模型（LMEMs）的显著性检验对HPO基准测试运行进行事后分析。LMEMs允许对整个实验数据进行灵活且富有表现力的建模，包括基准元特征等信息，从而提供比当前分析实践更深入的见解。我们通过对PriorBand论文实验数据的案例研究证明了这一点，发现了原始工作中未报告的见解。
英文摘要：
The importance of tuning hyperparameters in Machine Learning (ML) and Deep Learning (DL) is established through empirical research and applications, evident from the increase in new hyperparameter optimization (HPO) algorithms and benchmarks steadily added by the community. However, current benchmarking practices using averaged performance across many datasets may obscure key differences between HPO methods, especially for pairwise comparisons. In this work, we apply Linear Mixed-Effect Models-based (LMEMs) significance testing for post-hoc analysis of HPO benchmarking runs. LMEMs allow flexible and expressive modeling on the entire experiment data, including information such as benchmark meta-features, offering deeper insights than current analysis practices. We demonstrate this through a case study on the PriorBand paper's experiment data to find insights not reported in the original work.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
