arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2210.13587cond-mat.mtrl-sci
收藏

量化机器学习模型在材料发现中的性能

Quantifying the performance of machine learning models in materials discovery

Christopher K. H. Borg, Eric S. Muckley, Clara Nyby, James E. Saal, Logan Ward, Apurva Mehta, Bryce Meredig

AI总结:

本研究揭示传统静态误差指标与机器学习模型在序列学习引导的材料发现中的性能无明确关联,分析了影响发现性能的四类因素,并提出Discovery Yield和Discovery Probability作为更合适的评估指标。

摘要:

材料发现中使用的机器学习(ML)模型的预测能力通常采用简单统计量来衡量,例如ML预测的材料属性值与其已知值之间的均方根误差(RMSE)或决定系数($r^2$)。人们很容易做出这样的假设:低误差模型应该能有效指导材料发现,反之,高误差模型的发现性能会很差。然而,我们观察到,在整个训练集上平均得到的“静态”量(如RMSE),与ML属性模型动态指导具有目标属性的新型材料的迭代(且通常是外推性的)发现的能力之间,并不存在明确的关联。在本研究中,我们模拟了一个由序列学习(SL)引导的材料发现过程,并证明了传统模型误差指标与模型在指导材料发现时的性能之间存在解耦。我们发现,模型在材料发现中的性能很大程度上取决于:(1)属性分布中的目标范围(例如,是需要第一十分位还是第十十分位的材料);(2)SL采集函数中是否纳入了不确定性估计;(3)科研人员是关注单次发现还是多个目标;(4)允许进行多少次SL迭代。为了克服静态指标的局限性并稳健地衡量SL性能,我们推荐使用发现产率(Discovery Yield,$DY$)——衡量SL过程中发现的高性能材料数量的指标,以及发现概率(Discovery Probability,$DP$)——衡量在SL过程的任意阶段发现高性能材料的可能性的指标。

英文摘要:

The predictive capabilities of machine learning (ML) models used in materials discovery are typically measured using simple statistics such as the root-mean-square error (RMSE) or the coefficient of determination ($r^2$) between ML-predicted materials property values and their known values. A tempting assumption is that models with low error should be effective at guiding materials discovery, and conversely, models with high error should give poor discovery performance. However, we observe that no clear connection exists between a "static" quantity averaged across an entire training set, such as RMSE, and an ML property model's ability to dynamically guide the iterative (and often extrapolative) discovery of novel materials with targeted properties. In this work, we simulate a sequential learning (SL)-guided materials discovery process and demonstrate a decoupling between traditional model error metrics and model performance in guiding materials discoveries. We show that model performance in materials discovery depends strongly on (1) the target range within the property distribution (e.g., whether a 1st or 10th decile material is desired); (2) the incorporation of uncertainty estimates in the SL acquisition function; (3) whether the scientist is interested in one discovery or many targets; and (4) how many SL iterations are allowed. To overcome the limitations of static metrics and robustly capture SL performance, we recommend metrics such as Discovery Yield ($DY$), a measure of how many high-performing materials were discovered during SL, and Discovery Probability ($DP$), a measure of likelihood of discovering high-performing materials at any point in the SL process.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑