算法比较基准集的评价
On the Assessment of Benchmark Suites for Algorithm Comparison
摘要：
基准测试套件，即基准测试函数的集合，广泛用于黑盒优化算法的比较。多年来，研究已经确定了基准测试套件的许多理想品质，例如多样化的拓扑结构，不同的难度，可扩展性，现实世界问题的代表性等。然而，虽然拓扑结构的特点已经受到以前的研究，有没有研究统计评估的难度水平的基准函数，以及他们如何区分优化算法和如何适合是一个基准套件的算法比较。在本文中，我们提出了使用项目反应理论（IRT）模型，贝叶斯双参数逻辑模型的多次尝试，统计评估这些方面的经验成功率的算法。通过这个模型，我们可以评估每个基准测试的难度水平，它们区分不同算法的程度，算法的能力得分，以及基准测试套件在能力得分的估计中增加了多少信息。我们展示了使用这个模型在两个著名的基准套件，黑盒优化基准（BBOB）连续优化和伪布尔优化（PBO）离散优化。我们发现BBOB套件的大多数基准函数具有高难度（与优化算法相比）和低区分度。对于PBO，大多数函数都有很好的判别参数，但通常被认为太容易了。我们讨论了IRT在基准测试中的潜在用途，包括使用它来改进基准测试套件的设计，测量算法的多个方面，以及设计自适应套件。
英文摘要：
Benchmark suites, i.e. a collection of benchmark functions, are widely used in the comparison of black-box optimization algorithms. Over the years, research has identified many desired qualities for benchmark suites, such as diverse topology, different difficulties, scalability, representativeness of real-world problems among others. However, while the topology characteristics have been subjected to previous studies, there is no study that has statistically evaluated the difficulty level of benchmark functions, how well they discriminate optimization algorithms and how suitable is a benchmark suite for algorithm comparison. In this paper, we propose the use of an item response theory (IRT) model, the Bayesian two-parameter logistic model for multiple attempts, to statistically evaluate these aspects with respect to the empirical success rate of algorithms. With this model, we can assess the difficulty level of each benchmark, how well they discriminate different algorithms, the ability score of an algorithm, and how much information the benchmark suite adds in the estimation of the ability scores. We demonstrate the use of this model in two well-known benchmark suites, the Black-Box Optimization Benchmark (BBOB) for continuous optimization and the Pseudo Boolean Optimization (PBO) for discrete optimization. We found that most benchmark functions of BBOB suite have high difficulty levels (compared to the optimization algorithms) and low discrimination. For the PBO, most functions have good discrimination parameters but are often considered too easy. We discuss potential uses of IRT in benchmarking, including its use to improve the design of benchmark suites, to measure multiple aspects of the algorithms, and to design adaptive suites.
机构、总结、中文摘要均由AI分析而来，可能存在错误，请以论文正文为准。
