Analyzing the Impact of Undersampling on the Benchmarking and Configuration of Evolutionary Algorithms
Diederick Vermetten, Hao Wang, Manuel López-Ibañez, Carola Doerr, Thomas Bäck
备注：
To be published in proceedings of Genetic and Evolutionary Computation Conference (GECCO 22), July 9-13, 2022, Boston, MA, USA. ACM, New York, NY, USA, 9 pages. https://doi.org/10.1145/3512290.3528799
The stochastic nature of iterative optimization heuristics leads to inherently noisy performance measurements. Since these measurements are often gathered once and then used repeatedly, the number of collected samples will have a significant impact on the reliability of algorithm comparisons. We show that care should be taken when making decisions based on limited data. Particularly, we show that the number of runs used in many benchmarking studies, e.g., the default value of 15 suggested by the COCO environment, can be insufficient to reliably rank algorithms on well-known numerical optimization benchmarks.
Additionally, methods for automated algorithm configuration are sensitive to insufficient sample sizes. This may result in the configurator choosing a `lucky' but poor-performing configuration despite exploring better ones. We show that relying on mean performance values, as many configurators do, can require a large number of runs to provide accurate comparisons between the considered configurations. Common statistical tests can greatly improve the situation in most cases but not always. We show examples of performance losses of more than 20%, even when using statistical races to dynamically adjust the number of runs, as done by irace. Our results underline the importance of appropriately considering the statistical distribution of performance values.
EvoSampling: A Granular Ball-based Evolutionary Hybrid Sampling with Knowledge Transfer for Imbalanced Learning
School of Computer Science and Technology, Dalian University of Technology（大连理工大学计算机科学与技术学院）；Key Laboratory of Social Computing and Cognitive Intelligence (Dalian University of Technology), Ministry of Education（社会计算与认知智能重点实验室（大连理工大学））；School of Engineering and Computer Science, Victoria University of Wellington（惠灵顿维多利亚大学工程与计算机科学学院）；Department of Computer Science, Hong Kong Baptist University（香港浸会大学计算机科学系）
MToP: A MATLAB Benchmarking Platform for Evolutionary Multitasking
School of Computer Science, China University of Geosciences（中国地质大学计算机学院）；Trustworthy and General Artificial Intelligence Laboratory, School of Engineering, Westlake University（西湖大学工学院可信与通用人工智能实验室）；College of Systems Engineering, National University of Defense Technology（国防科技大学系统工程学院）；School of Computer Engineering, Hubei University of Arts and Science（湖北文理学院计算机工程学院）；College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院）；Centre for Frontier AI Research (CFAR), Agency for Science, Technology and Research（新加坡科技研究局前沿人工智能研究中心）
Benchmarking that Matters: Rethinking Benchmarking for Practical Impact
Leiden University（莱顿大学）；Hochschule des Bundes für öffentliche Verwaltung（联邦公共管理高等学校）；TH Köln（科隆应用技术大学）；Ruhr-Universität Bochum（波鸿鲁尔大学）；Vorarlberg University of Applied Sciences（福拉尔贝格应用科学大学）；GWDG（哥廷根国家研究与应用数据中心）；Brno University of Technology（布尔诺理工大学）；University of Münster（明斯特大学）；University of Stirling（斯特灵大学）；Paderborn University（帕德博恩大学）；Sorbonne Université（索邦大学）；West University of Timișoara（蒂米什瓦拉西部大学）
ERNAS: An Evolutionary Neural Architecture Search for Magnetic Resonance Image Reconstructions
Gordon Center for Medical Imaging, Massachusetts General Hospital and Harvard Medical School（马萨诸塞总医院与哈佛医学院戈登医学影像中心）；Texas A&M University（德克萨斯A&M大学）；Texas A&M Institute of Data Science, Texas A&M University（德克萨斯A&M大学数据科学研究所）；Texas A&M University at Qatar（德克萨斯A&M大学卡塔尔分校）
Block-Bench: A Framework for Controllable and Transparent Discrete Optimization Benchmarking
LIACS, Leiden University（莱顿大学）；Optimisation and Logistics, School of Computer Science and Information Technology, Adelaide University（阿德莱德大学）；Adelaide University（阿德莱德大学）
Large Language Models and Evolutionary Computation: A Critical Review of Bidirectional Interaction, Automated Algorithm Design, and Co-Adaptive Systems
Department of Electrical and Computer Engineering, National University of Singapore（新加坡国立大学电气与计算机工程系）；Department of Computer Science, Universidad de Jaén（西班牙Jaén大学计算机科学系）；School of Computer and Mathematical Sciences, University of Adelaide（阿德莱德大学计算机与数学科学学院）；Leiden Institute of Advanced Computer Science, University Leiden（莱顿大学先进计算机科学研究所）；Department of Mathematics and Computing, Dr. B. R. Ambedkar National Institute of Technology（德拉·B·R·阿姆贝卡尔国立理工学院数学与计算系）
Theory-inspired Parameter Control Benchmarks for Dynamic Algorithm Configuration
University of Freiburg（弗赖堡大学）；University of St Andrews（圣安德鲁斯大学）；Sorbonne Université（索邦大学）；CNRS（法国国家科学研究中心）；Bosch Center for Artificial Intelligence（博世人工智能中心）
TECNALIA（泰克纳利亚研究院）；University of the Basque Country (UPV/EHU)（巴斯克地区大学）；DaSCI Andalusian Institute of Data Science and Computational Intelligence（安达卢西亚数据科学与计算智能研究所）；University of Granada（格拉纳达大学）
Evolving Hard Maximum Cut Instances for Quantum Approximate Optimization Algorithms
LIACS, Leiden University（莱顿大学信息科学研究中心）；Optimisation and Logistics, School of Computer and Mathematical Sciences, The University of Adelaide（计算机与数学科学学院优化与物流系，阿德莱德大学）；Mathematical Sciences, The University of Adelaide（数学科学，阿德莱德大学）
