Benchmarking in Optimization: Best Practice and Open Issues

The Design and Implementation of a Scalable DL Benchmarking Platform

The current Deep Learning (DL) landscape is fast-paced and is rife with non-uniform models, hardware/software (HW/SW) stacks, but lacks a DL benchmarking platform to facilitate evaluation and comparison of DL innovations, be it models, frameworks, libraries, or hardware. Due to the lack of a benchmarking platform, the current practice of evaluating the benefits of proposed DL innovations is both arduous and error-prone - stifling the adoption of the innovations. In this work, we first identify $10$ design features which are desirable within a DL benchmarking platform. These features include: p…

聚类分析中的基准测试：一份白皮书

Benchmarking in cluster analysis: A white paper

本文针对无监督聚类分析中基准测试指南匮乏的问题，利用模拟与实证数据探讨其理论基础，并提出聚类基准测试的实践操作细节与基础性建议。

Data-driven surrogate modelling and benchmarking for process equipment

In chemical process engineering, surrogate models of complex systems are often necessary for tasks of domain exploration, sensitivity analysis of the design parameters, and optimization. A suite of computational fluid dynamics (CFD) simulations geared toward chemical process equipment modeling has been developed and validated with experimental results from the literature. Various regression-based active learning strategies are explored with these CFD simulators in-the-loop under the constraints of a limited function evaluation budget. Specifically, five different sampling strategies and five r…

有意义的基准测试：重新思考具有实际影响的基准测试

Benchmarking that Matters: Rethinking Benchmarking for Practical Impact

针对进化计算领域当前基准测试实践与现实需求脱节的问题，本文指出其关键差距，提出构建受现实启发的基准、可访问特征空间及社区维护性能数据库的愿景，以推动科学与工业应用协同发展。

弥合理论与实践之间的鸿沟：迁移进化优化的基准测试

Bridging the Gap Between Theory and Practice: Benchmarking Transfer Evolutionary Optimization

本文提出一个基于数量、多样性和速度三维度的实用迁移进化优化基准套件，旨在评估现有算法并推动新方法发展，以弥合理论与实践差距。

Benchmarking Feature-based Algorithm Selection Systems for Black-box Numerical Optimization

Feature-based algorithm selection aims to automatically find the best one from a portfolio of optimization algorithms on an unseen problem based on its landscape features. Feature-based algorithm selection has recently received attention in the research field of black-box numerical optimization. However, there is still room for analysis of algorithm selection for black-box optimization. Most previous studies have focused only on whether an algorithm selection system can outperform the single-best solver in a portfolio. In addition, a benchmarking methodology for algorithm selection systems has…

Leveraging Benchmarking Data for Informed One-Shot Dynamic Algorithm Selection

A key challenge in the application of evolutionary algorithms in practice is the selection of an algorithm instance that best suits the problem at hand. What complicates this decision further is that different algorithms may be best suited for different stages of the optimization process. Dynamic algorithm selection and configuration are therefore well-researched topics in evolutionary computation. However, while hyper-heuristics and parameter control studies typically assume a setting in which the algorithm needs to be chosen while running the algorithms, without prior information, AutoML app…

无人机路径规划的全局优化技术基准测试

Benchmarking global optimization techniques for unmanned aerial vehicle path planning

本文提出用无人机路径规划问题作为全局优化算法的基准测试，生成56个实例，通过比较十二种算法，发现进化算法表现最佳，并指出可变维度特征待深入研究。

Black-box optimization benchmarking of IPOP-saACM-ES and BIPOP-saACM-ES on the BBOB-2012 noiseless testbed

In this paper, we study the performance of IPOP-saACM-ES and BIPOP-saACM-ES, recently proposed self-adaptive surrogate-assisted Covariance Matrix Adaptation Evolution Strategies. Both algorithms were tested using restarts till a total number of function evaluations of $10^6D$ was reached, where $D$ is the dimension of the function search space. We compared surrogate-assisted algorithms with their surrogate-less versions IPOP-saACM-ES and BIPOP-saACM-ES, two algorithms with one of the best overall performance observed during the BBOB-2009 and BBOB-2010. The comparison shows that the surrogate-a…

Best Practices for Applying Deep Learning to Novel Applications

This report is targeted to groups who are subject matter experts in their application but deep learning novices. It contains practical advice for those interested in testing the use of deep neural networks on applications that are novel for deep learning. We suggest making your project more manageable by dividing it into phases. For each phase this report contains numerous recommendations and insights to assist novice practitioners.

混合整数黑箱优化中差分进化参数控制方法的基准测试

Benchmarking Parameter Control Methods in Differential Evolution for Mixed-Integer Black-Box Optimization

本文在bbob-mixint套件上组件化评测DE参数控制方法，发现最佳PCM取决于变异与修复组合，简单CoDE式PCM通常优于SHADE，且合适PCM使DE在大预算下胜过整数处理版CMA-ES。

A Comprehensive Review of Spiking Neural Networks: Interpretation, Optimization, Efficiency, and Best Practices

Biological neural networks continue to inspire breakthroughs in neural network performance. And yet, one key area of neural computation that has been under-appreciated and under-investigated is biologically plausible, energy-efficient spiking neural networks, whose potential is especially attractive for low-power, mobile, or otherwise hardware-constrained settings. We present a literature review of recent developments in the interpretation, optimization, efficiency, and accuracy of spiking neural networks. Key contributions include identification, discussion, and comparison of cutting-edge met…