arXivDaily arXiv每日学术速递 周一至周五更新
arXiv 2310.16802cs.LG
收藏

从分子到材料:面向原子性质预测的大型可泛化模型预训练

From Molecules to Materials: Pre-training Large Generalizable Models for Atomic Property Prediction

Carnegie Mellon University(卡内基梅隆大学);Fundamental AI Research (FAIR) at Meta(Meta基础AI研究(FAIR))

Nima Shoghi, Adeesh Kolluru, John R. Kitchin, Zachary W. Ulissi, C. Lawrence Zitnick, Brandon M. Wood

AI总结:

本文提出联合多领域预训练策略 JMP,在约 1.2 亿个体系的多化学领域数据上同时训练,并在多个下游任务中平均提升 59%,在 40 个任务中的 34 个达到或刷新最优结果,展示了多样化数据预训练对跨领域原子性质预测的价值。

摘要:

基础模型在自然语言处理和计算机视觉等机器学习领域已带来变革性影响。然而,由于跨多个化学领域训练有效模型存在挑战,类似成功在原子性质预测中仍较为有限。为解决这一问题,我们提出联合多领域预训练(Joint Multi-domain Pre-training,JMP),这是一种有监督预训练策略,在多个来自不同化学领域的数据集上同时训练,将每个数据集视为多任务框架中的独特预训练任务。我们的组合训练数据集包含来自 OC20、OC22、ANI-1x 和 Transition-1x 的约 1.2 亿个体系。我们通过在 QM9、rMD17、MatBench、QMOF、SPICE 和 MD22 等多样化下游任务和数据集上进行微调来评估性能和泛化能力。JMP 相较于从头训练平均提升 59%,并在 40 个任务中的 34 个上达到或刷新最优结果。我们的工作凸显了利用多样化数据的预训练策略在推进跨化学领域性质预测方面的潜力,尤其适用于低数据任务。更多信息请访问 https://nima.sh/jmp。

英文摘要:

Foundation models have been transformational in machine learning fields such as natural language processing and computer vision. Similar success in atomic property prediction has been limited due to the challenges of training effective models across multiple chemical domains. To address this, we introduce Joint Multi-domain Pre-training (JMP), a supervised pre-training strategy that simultaneously trains on multiple datasets from different chemical domains, treating each dataset as a unique pre-training task within a multi-task framework. Our combined training dataset consists of $\sim$120M systems from OC20, OC22, ANI-1x, and Transition-1x. We evaluate performance and generalization by fine-tuning over a diverse set of downstream tasks and datasets including: QM9, rMD17, MatBench, QMOF, SPICE, and MD22. JMP demonstrates an average improvement of 59% over training from scratch, and matches or sets state-of-the-art on 34 out of 40 tasks. Our work highlights the potential of pre-training strategies that utilize diverse data to advance property prediction across chemical domains, especially for low-data tasks. Please visit https://nima.sh/jmp for further information.

机构、总结、中文摘要均由AI分析而来,可能存在错误,请以论文正文为准。

↑