arXivDaily arXiv每日学术速递 周一至周五更新
arXiv周末暂无论文更新,休息一下吧,周末愉快~~
arXiv 2204.11527cs.NE
收藏

SELECTOR:选择代表性基准套件进行可重现的统计比较

SELECTOR: Selecting a Representative Benchmark Suite for Reproducible Statistical Comparison

Gjorgjina Cenikj, Ryan Dieter Lang, Andries Petrus Engelbrecht, Carola Doerr, Peter Korošec, Tome Eftimov

摘要:

公平算法评估的条件是存在高质量的基准数据集,这些数据集是非冗余的,并且代表了典型的优化场景。在本文中,我们评估了三个选择不同的问题实例,应参与优化算法的比较,以确保强大的统计算法的性能分析。第一种方法采用聚类来识别类似的问题实例组和随后的采样从每个集群构建新的基准,而其他两种方法使用图形算法来识别支配和最大独立的节点集。我们证明了所提出的算法的适用性进行统计性能分析的五个投资组合组成的三个最常用的优化基准的优化算法。结果表明,在每个基准上分别进行的算法性能的统计分析产生了相互矛盾的结果,这些结果可以用来给出一个算法优于另一个算法的错误指示。另一方面,当对使用所提出的分类法选择的问题实例进行分析时,这些问题实例均匀地覆盖问题景观,统计结果是稳健和一致的。

英文摘要:

Fair algorithm evaluation is conditioned on the existence of high-quality benchmark datasets that are non-redundant and are representative of typical optimization scenarios. In this paper, we evaluate three heuristics for selecting diverse problem instances which should be involved in the comparison of optimization algorithms in order to ensure robust statistical algorithm performance analysis. The first approach employs clustering to identify similar groups of problem instances and subsequent sampling from each cluster to construct new benchmarks, while the other two approaches use graph algorithms for identifying dominating and maximal independent sets of nodes. We demonstrate the applicability of the proposed heuristics by performing a statistical performance analysis of five portfolios consisting of three optimization algorithms on five of the most commonly used optimization benchmarks. The results indicate that the statistical analyses of the algorithms' performance, conducted on each benchmark separately, produce conflicting outcomes, which can be used to give a false indication of the superiority of one algorithm over another. On the other hand, when the analysis is conducted on the problem instances selected with the proposed heuristics, which uniformly cover the problem landscape, the statistical outcomes are robust and consistent.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑