Benchmarking studies in computational chemistry use reference datasets to assess the accuracy of a method through error statistics. The commonly used error statistics, such as the mean signed and mean unsigned errors, do not inform end-users on the expected amplitude of prediction errors attached to these methods. We show that, the distributions of model errors being neither normal nor zero-centered, these error statistics cannot be used to infer prediction error probabilities. To overcome this limitation, we advocate for the use of more informative statistics, based on the empirical cumulative distribution function of unsigned errors, namely (1) the probability for a new calculation to have an absolute error below a chosen threshold, and (2) the maximal amplitude of errors one can expect with a chosen high confidence level. Those statistics are also shown to be well suited for benchmarking and ranking studies. Moreover, the standard error on all benchmarking statistics depends on the size of the reference dataset. Systematic publication of these standard errors would be very helpful to assess the statistical reliability of benchmarking conclusions.
Probabilistic performance estimators for computational chemistry methods: Systematic Improvement Probability and Ranking Probability Matrix. II. Applications
Stochastic tensor contraction for quantum chemistry
Division of Chemistry and Chemical Engineering, California Institute of Technology(加州理工学院化学与化学工程系);Marcus Center for Theoretical Chemistry, California Institute of Technology(加州理工学院马库斯理论化学中心)
Scaling Neural Network Quantum States for Ab Initio Quantum Chemistry
State Key Lab of Brain-Machine Intelligence, Zhejiang University(浙江大学脑机智能全国重点实验室);College of Artificial Intelligence, Zhejiang University(浙江大学人工智能学院);State Key Laboratory of Precision and Intelligent Chemistry, University of Science and Technology of China(中国科学技术大学精密与智能化学全国重点实验室)
$Δ$-ML Ensembles for Selecting Quantum Chemistry Methods to Compute Intermolecular Interactions
School of Chemistry and Biochemistry(化学与生物化学系);Georgia Institute of Technology(佐治亚理工学院);Center for Artificial Intelligence in Science and Engineering(科学与工程中的人工智能中心);Center for Computational Molecular Science and Technology(计算分子科学与技术中心);School of Computational Science and Engineering(计算科学与工程系)
Benchmarking Basis Sets for Density Functional Theory Thermochemistry Calculations: Why unpolarised basis sets and the polarised 6-311G family should be avoided